Forem: Dash0

OpenTelemetry Filelog Receiver: A Guide to Ingesting Log Files

Ayooluwa Isaiah — Tue, 02 Dec 2025 13:38:46 +0000

Even in the age of cloud-native apps and distributed tracing, plain old log files remain one of the richest sources of truth in any system. From legacy business applications and batch jobs to NGINX, databases, and on-prem infrastructure, critical diagnostics still end up written to disk.

The OpenTelemetry Collector filelog receiver gives you a way to bring those logs into a modern observability pipeline. It continuously tails files, parses their contents, and converts raw text into structured OpenTelemetry LogRecords.

This guide shows you how to put that power to work, from the basics of reading a file to building a production-ready pipeline that handles rotation, recovers from restarts, and never loses a single line. You'll learn how to structure, enrich, and standardize log file entries so they become first-class observability data.

Let's begin!

How the Filelog receiver works

Before we get into configuration details, it helps to picture how the receiver handles a log file throughout its lifecycle. You can think of it as a simple repeating four-step loop:

Discover: The receiver scans the filesystem at regular intervals, using the include and exclude patterns you've set, to figure out which log files it should pay attention to.
Read: Once a file is picked up, the receiver opens it and begins following along as new lines are written. The start_at setting decides whether it begins from beginning or just tails new content from the end.
Parse: Each line (or block of lines, if multiline parsing is used) runs through a series of Stanza operators (if configured). These operators parse the raw text, pull out key attributes, assign timestamps and severity levels, and ultimately structure the log data.
Emit: Finally, the structured log records are passed into the Collector's pipeline, where they can be filtered, transformed further, or exported to your backend.

This Discover -> Read -> Parse -> Emit loop forms the foundation of everything the receiver does.

Quick Start: tailing a log file

One of the most common use cases is when your application is already writing logs in JSON format to a file. For example, imagine you have a service writing JSON logs to /var/log/myapp/app.log:

{"time":"2025-09-28 20:15:12","level":"INFO","message":"User logged in successfully","user_id":"u-123","source_ip":"192.168.1.100"}
{"time":"2025-09-28 20:15:45","level":"WARN","message":"Password nearing expiration","user_id":"u-123"}

Here's a minimal filelog receiver example to read and ingest such logs into an OpenTelemetry pipeline:

# otelcol.yaml
receivers:
  filelog:
    # 1. DISCOVER all .log files in /var/log/myapp/
    include: [/var/log/myapp/*.log]
    # 2. READ from the beginning of new files
    start_at: beginning
    # 3. PARSE using the json_parser operator
    operators:
      - type: json_parser
        # Tell the parser where to find the timestamp and how it's formatted
        timestamp:
          parse_from: attributes.time
          layout: "%Y-%m-%d %H:%M:%S"
        # Tell the parser which field contains the severity
        severity:
          parse_from: attributes.level

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [debug]

Here's a breakdown of the above configuration:

include: Points the receiver to all .log files in /var/log/myapp/.
start_at: beginning: Ensures the receiver processes the entire file the first time it sees it. By default (end), it would only capture new lines written after the Collector starts.
operators: In this case, there's just one: the json_parser. Its job is to take each log line, interpret it as JSON, and then promote selected fields into the log record's core metadata.
timestamp and severity: Within the json_parser, we're pulling the time and level fields out of the JSON and promoting them to the OpenTelemetry's top-level Timestamp and Severity* fields for each log record.

With the debug exporter, you'll see the parsed and structured output. Instead of just raw JSON, each field is now properly represented inside each log record:

LogRecord #0
ObservedTimestamp: 2025-09-28 20:48:36.728437503 +0000 UTC
Timestamp: 2025-09-28 20:15:12 +0000 UTC
SeverityText: INFO
SeverityNumber: Info(9)
Body: Str({"time":"2025-09-28 20:15:12","level":"INFO","message":"User logged in successfully","user_id":"u-123","source_ip":"192.168.1.100"})
Attributes:
     -> user_id: Str(u-123)
     -> source_ip: Str(192.168.1.100)
     -> log.file.name: Str(myapp.log)
     -> time: Str(2025-09-28 20:15:12)
     -> level: Str(INFO)
     -> message: Str(User logged in successfully)
Trace ID:
Span ID:
Flags: 0

The raw JSON logs have now been converted into OpenTelemetry's unified log data format, ensuring a consistent foundation for cross-system observability.

The log.file.name attribute is automatically added by the receiver by default, and you can also enable include_file_path to capture the full file path as well:

# otelcol.yaml
receivers:
  filelog:
    include: [/var/log/myapp/*.log]
    include_file_path: true

This allows you to easily filter or query logs based on their exact source path:

Attributes:
     -> log.file.path: Str(/var/log/myapp/app.log)
     -> log.file.name: Str(app.log)

You can find more enrichment options in the official OpenTelemetry Filelog receiver documentation.

Filtering and managing log files

The most fundamental step in configuring the filelog receiver is telling it which files to monitor. This is controlled using include and exclude glob patterns.

The receiver first uses include to generate a list of all potential files, then it applies the exclude patterns to remove any unwanted files from that list.

Here's an example:

# otelcol.yaml
receivers:
  filelog:
    include: [/var/log/apps/**/*.log]
    exclude:
      - /var/log/apps/**/debug.log
      - /var/log/apps/**/*.tmp

In this scenario, the receiver will collect every .log file under /var/log/apps/, including subdirectories, but it will skip any file named debug.log and any file ending with .tmp.

Excluding files by modification age

If the log directory you're reading contains many existing log files, you can instruct the receiver to ignore files that have not been modified within a given time window with exclude_older_than:

# otelcol.yaml
receivers:
  filelog:
    include: [/var/log/myapp/*.log]
    exclude_older_than: 24h
    start_at: beginning

In this example, even if app-2025-07-15.log matches the pattern, it will be skipped if it hasn't been updated in the past 24 hours.

Parsing unstructured text with regular expressions

Most infrastructure logs don't come neatly packaged as JSON. More often, they're plain text strings that follow a loose pattern, such as web server access logs, database query logs, or operating system messages. These logs are human-readable but difficult for machines to analyze until they're given some structure.

The Collector addresses this with the regex_parser operator. Using regular expressions with named capture groups, you can break a raw log line into meaningful fields and promote them into structured attributes.

For example, consider an NGINX access log in the Common Log Format:

127.0.0.1 - - [28/Sep/2025:20:30:00 +0000] "GET /api/v1/users HTTP/1.1" 200 512
127.0.0.1 - - [28/Sep/2025:20:30:05 +0000] "POST /api/v1/login HTTP/1.1" 401 128

You can configure the regex_parser like this to parse them into structured attributes:

# otelcol.yaml
receivers:
  filelog:
    include: [/var/log/nginx/access.log]
    start_at: beginning
    operators:
      - type: regex_parser
        # Use named capture groups to extract data
        regex:
          '^(?P<client_ip>[^ ]+) - - \[(?P<timestamp>[^\]]+)\]
          "(?P<http_method>[A-Z]+) (?P<http_path>[^ "]+)[^"]*"
          (?P<status_code>\d{3}) (?P<response_size>\d+)$'
        # Parse the extracted timestamp
        timestamp:
          parse_from: attributes.timestamp
          layout: "%d/%b/%Y:%H:%M:%S %z"
        # Map status codes to severities
        severity:
          parse_from: attributes.status_code
          mapping:
            info:
              - min: 200
                max: 399
            warn: 4xx
            error: 5xx

The core of this setup is the regex expression with named capture groups. Each group labels a slice of the line so the parser can turn it into an attribute: client_ip grabs the remote address, timestamp captures the bracketed time string, http_method and http_path pull the request pieces, status_code picks up the three-digit response code, and response_size records the byte count.

Once those attributes exist, the timestamp field parses the timestamp string into a proper datetime value, and the severity block translates status codes into meaningful severity levels using an explicit mapping: 2xx and 3xx responses as INFO, 4xx as WARN, and 5xx as ERROR.

Once access logs are ingested with this configuration, you'll see a structured log record with all the important pieces extracted out as attributes:

LogRecord #0
ObservedTimestamp: 2025-09-28 21:17:42.31729069 +0000 UTC
Timestamp: 2025-09-28 20:30:00 +0000 UTC
SeverityText: 200
SeverityNumber: Info(9)
Body: Str(127.0.0.1 - - [28/Sep/2025:20:30:00 +0000] "GET /api/v1/users HTTP/1.1" 200 512)
Attributes:
     -> status_code: Str(200)
     -> response_size: Str(512)
     -> log.file.name: Str(myapp.log)
     -> client_ip: Str(127.0.0.1)
     -> timestamp: Str(28/Sep/2025:20:30:00 +0000)
     -> http_method: Str(GET)
     -> http_path: Str(/api/v1/users)
Trace ID:
Span ID:
Flags: 0

With a single expression and a couple of parsing steps, a flat NGINX access log is transformed into structured OpenTelemetry data. A natural next step is aligning the captured attributes with the HTTP semantic conventions through the attributes processor or transform processor.

Handling multiple log formats

Log files rarely come in just one flavor. For example, you might be ingesting NGINX logs, database logs, and application logs, each with their own format.

The cleanest way to handle this is to define a separate filelog receiver for each file type. Each receiver has its own parsing rules and runs independently, which keeps your setup organized and easy to debug.

This is the best approach when the log formats are completely different and share nothing in common.

# otelcol.yaml
receivers:
  # NGINX access logs
  filelog/nginx_access:
    include: [/var/log/nginx/access.log]
    operators:
      - type: regex_parser
        # ... NGINX access log parsing rules

  # NGINX error logs
  filelog/nginx_error:
    include: [/var/log/nginx/error.log]
    operators:
      - type: regex_parser
        # ... NGINX error log parsing rules

Sometimes, though, variation happens within a single file.

Maybe most lines are simple messages, but others add extra fields like a trace_id:

INFO: Application started successfully.
DEBUG: Processing request for trace_id=12345

Instead of writing one massive regex to cover every case, you can use conditional operators with if:

# otelcol.yaml
receivers:
  filelog:
    include: [/var/log/app.log]
    operators:
      # Parse the basic structure of every line
      - type: regex_parser
        id: base_parser # a unique ID is required when multiple operators of the same type is being used
        regex: '^(?P<severity>\w+): (?P<message>.*)$'

      # Only run this parser when "trace_id" appears
      - type: regex_parser
        id: trace_parser
        if: 'attributes["message"] matches "trace_id"'
        parse_from: attributes.message
        regex: '.*trace_id=(?P<trace_id>\w+).*'

Here's what happens:

The first parser runs on every log line and extracts severity and message.
The second parser runs only when the message contains trace_id, enriching the log with that extra field.

By combining these two approaches, multiple receivers for unrelated formats and conditional parsing for minor variations, you can handle almost any kind of log your systems produce without creating unreadable or brittle configurations.

Handling stack traces and multiline logs

Not all log entries fit neatly on a single line. A stack trace is a classic example:

2025-09-28 21:05:42 [ERROR] Unhandled exception: Cannot read property 'foo' of undefined
TypeError: Cannot read property 'foo' of undefined
    at Object.<anonymous> (/usr/src/app/index.js:15:18)
    at Module._compile (node:internal/modules/cjs/loader:1254:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
    at Module.load (node:internal/modules/cjs/loader:1117:32)
    at Module._load (node:internal/modules/cjs/loader:958:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
    at node:internal/main/run_main_module:17:47

If you send this directly to the Collector, the filelog receiver will treat each line as a separate log record. That's not what you want, since the error message and every stack frame belong together.

The fix is to use the multiline configuration, which tells the receiver how to group lines into a single entry:

# otelcol.yaml
receivers:
  filelog:
    include: [/var/log/myapp/*.log]
    start_at: beginning

    multiline:
      # New entry starts when a line begins with "YYYY-MM-DD HH:MM:SS"
      line_start_pattern: ^\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}

    operators:
      - type: regex_parser
        regex: (?P<timestamp>\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2})\s+\[(?P<severity>[A-Za-z]+)\]\s+(?P<message>.+)

        timestamp:
          parse_from: attributes.timestamp
          layout: "%Y-%m-%d %H:%M:%S"

        severity:
          parse_from: attributes.severity

Here, the line_start_pattern acts as the anchor. A new log entry begins only when a line starts with a date in the form YYYY-MM-DD HH:MM:SS, and any line that doesn't match is appended to the previous one.

The result is that the entire stack trace, from the error message down through each at ... frame, gets captured as one structured log record. This preserves full context, making it far easier to analyze and troubleshoot errors.

LogRecord #0
ObservedTimestamp: 2025-10-07 12:04:26.963143642 +0000 UTC
Timestamp: 2025-09-28 21:05:42 +0000 UTC
SeverityText: ERROR
SeverityNumber: Error(17)
Body: Str(2025-09-28 21:05:42 [ERROR] Unhandled exception: Cannot read property 'foo' of undefined
TypeError: Cannot read property 'foo' of undefined
    at Object.<anonymous> (/usr/src/app/index.js:15:18)
    at Module._compile (node:internal/modules/cjs/loader:1254:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
    at Module.load (node:internal/modules/cjs/loader:1117:32)
    at Module._load (node:internal/modules/cjs/loader:958:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
    at node:internal/main/run_main_module:17:47)
Attributes:
     -> log.file.name: Str(/var/log/myapp/app.log)
     -> message: Str(Unhandled exception: Cannot read property 'foo' of undefined)
     -> timestamp: Str(2025-09-28 21:05:42)
     -> severity: Str(ERROR)
Trace ID:
Span ID:
Flags: 0

Parsing metadata from file headers

Some log files don't just contain log entries. They begin with a header section that holds important metadata about the entire file. Without that context, the individual log lines can be hard to interpret.

This pattern is common with batch jobs and export processes. For example, a nightly billing run might write a fresh log file for each execution. At the top of that file you might see something like this:

# Job-ID: job-d8e8fca2
# Job-Type: nightly-billing-run
# Executed-By: scheduler-prod-1
# Records-To-Process: 1500
2025-10-08T08:20:00Z INFO: Starting billing run.
2025-10-08T08:21:15Z INFO: Processed account #1.
2025-10-08T08:21:16Z WARN: Account #2 has a negative balance.
. . .

Those first lines tell you exactly which job produced the logs that follow. If you ignore them, you lose that crucial context. The header feature solves this by parsing metadata from the top of the file and stamping it onto every subsequent log record.

It defines a small, dedicated pipeline that runs only on the initial block of lines. You need to specify a regex to match which lines belong to the header. The metadata_operators then parse those lines into attributes which are automatically added to every log entry that follows.

To use this feature, you need to do three things:

Enable the filelog.allowHeaderMetadataParsing feature gate:

# docker-compose.yml
services:
  otelcol:
    command:
      [
        --config=/etc/otelcol-contrib/config.yaml,
        --feature-gates=filelog.allowHeaderMetadataParsing,
      ]

Set start_at: beginning since the header has to be read from the top.
Configure both the header rules and the main operators pipeline.

Here's the configuration to parse the headers in the sample log file:

# otelcol.yaml
receivers:
  filelog:
    include: [/var/log/jobs/*.log]
    start_at: beginning # required
    header:
      pattern: ^#
      metadata_operators:
        - type: key_value_parser
          delimiter: ": "
          pair_delimiter: "# "

Here's what's happening:

pattern: ^# says that any line starting with # belongs to the header. Those header lines are then passed through the pipeline of metadata_operators.
The key_value_parser operator splits each header line into a key and value using : as the separator, while # denotes the beginning of a new key/value pair.

These results in the following attributes on every log entry that follows in that file:

Attributes:
     -> Job-ID: Str(job-d8e8fca2)
     -> Job-Type: Str(nightly-billing-run)
     -> Executed-By: Str(scheduler-prod-1)
     -> Records-To-Process: Str(1500)

As you can see, the Job-ID and other header fields are now attached to the log record, providing invaluable context that would have otherwise been lost.

From here, you can process them further by promoting the header fields to resource attributes and aligning with OpenTelemetry semantic conventions.

How to avoid lost or duplicate logs

When the Collector restarts, log ingestion can easily go wrong if state is not preserved as you risk either re-ingesting old data or skipping over new logs. If you use start_at: beginning, the receiver will reread all your log files and create massive duplication. With start_at: end, you might miss any entries written while the Collector was down.

The way to solve this is with checkpointing. By configuring a storage extension, you instruct the filelog receiver to save its position in each file (the last read offset) to disk and pick up exactly where it left off.

A conventional approach is using the file_storage extension for this purpose:

# otelcol.yaml
extensions:
  file_storage:
    directory: /var/otelcol/storage

receivers:
  filelog:
    include: [/var/log/myapp/*.log]
    start_at: beginning
    # Link the receiver to the storage extension
    storage: file_storage

# ... processors, exporters

service:
  # The extension must be enabled in the service section
  extensions: [file_storage]
  pipelines:
    logs:
      receivers: [filelog]
      # ...

With the storage extension enabled, the receiver will:

On startup, check the /var/otelcol/storage directory for saved offsets.
Resume reading from the saved offset for any file it was tracking, ensuring no data is lost or duplicated.
Periodically update the storage with its latest progress.

Checkpointing ensures that log collection is resilient to restarts, upgrades, and even crashes. It is a critical best practice for reliable log ingestion.

Handling log delivery failures gracefully

Checkpointing with a storage extension protects you during Collector restarts, but another common failure mode is when the receiver reads a batch successfully but fails to hand it off to the next stage.

This can happen if an exporter can't reach its endpoint, or the memory limiter is refusing data. By default, the receiver will drop that batch of logs and move on to the next, causing silent data loss.

To prevent this, the receiver has a built-in mechanism to retry sending failed batches. When retry_on_failure is enabled, the receiver will pause, wait for a configured interval, and attempt to resend the exact same batch of logs. This process repeats with an exponential backoff until the batch is sent successfully or the max_elapsed_time is reached:

# otelcol.yaml
receivers:
  filelog:
    retry_on_failure:
      enabled: true
      # Wait 5 seconds after the first failure before the first retry.
      initial_interval: 5s
      # The longest the receiver will wait between retries is 30 seconds.
      max_interval: 30s
      # Give up trying to send a batch after 10 minutes.
      max_elapsed_time: 10m

By combining checkpointing with a robust retry policy , you'll create a highly resilient log file ingestion pipeline that can withstand both Collector restarts and temporary downstream outages or throttling.

Deleting log files after processing

Some workflows call for processing a file once and then removing it to save space and avoid reprocessing. You can enable this with delete_after_read, which requires start_at: beginning:

# otelcol.yaml
receivers:
  filelog:
    include: [/var/log/archives/*.gz]
    start_at: beginning
    delete_after_read: true

You must need also enable the filelog.allowFileDeletion feature gate for this to work:

# docker-compose.yml
services:
  otelcol:
    command:
      [
        --config=/etc/otelcol-contrib/config.yaml,
        --feature-gates=filelog.allowFileDeletion,
      ]

Finally, ensure that the files are configured to be deletable and that the Collector service has enough permissions to delete the file. If permissions are insufficient, you will see a "could not delete" log record:

2025-10-08T06:42:03.973Z        error   reader/reader.go:278    could not delete        {"resource": {"service.instance.id": "7c0daf0e-e625-4da8-9577-072606dce057", "service.name": "otelcol-contrib", "service.version": "0.136.0"}, "otelcol.component.id": "filelog", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "component": "fileconsumer", "path": "/var/log/myapp/app.log", "filename": "/var/log/myapp/app.log"}

Just be careful when enabling this setting as it deletes the files from disk permanently.

Handling log rotation seamlessly

Log files don't grow indefinitely. Eventually, they'll get rotated (or at least they should). The filelog receiver is built to handle common rotation patterns, such as app.log to app.log.1 automatically and without losing data.

Instead of relying on filenames alone, the receiver tracks each file using a unique fingerprint derived from the first few kilobytes of content. When rotation occurs, it recognizes that the original file has been renamed, finishes reading it, and then starts fresh from the beginning of the new app.log.

This behavior requires no additional configuration; it works out of the box, giving you reliable log ingestion even in environments with frequent rotations.

Reading compressed files

Many log rotation tools compress old logs to save disk space, producing files like access.log.1.gz. The filelog receiver can handle these seamlessly by decompressing them on the fly.

To make this work, you use the compression setting. This tells the receiver that some or all of the files it discovers may be compressed and need to be decompressed before parsing.

You have two main choices for the compression setting:

gzip: Treats all matched files as gzip-compressed.
auto: Automatically detects compression based on file extension (currently .gz). This is the best option when a directory contains a mix of active, uncompressed logs and older, compressed ones.

For example, if your directory has both app.log (active) and app.log.1.gz (rotated and compressed), you can configure the receiver like this:

# otelcol.yaml
receivers:
  filelog:
    include: [/var/log/myapp/*]
    start_at: beginning
    # Automatically detect and decompress .gz files
    compression: auto
    operators:
      - type: regex_parser
        # ... your parsing rules

When working with compressed logs, there are two main things to keep in mind.

First, the receiver assumes that compressed files can only grow by appending new data. If a file is completely rewritten, for example by taking the original content and recompressing it together with new lines, the receiver may not handle it correctly.

Second, there's the question of fingerprinting. By default, the receiver identifies files based on their compressed bytes. This works fine in most cases, but if files are renamed or moved it can cause confusion. To make identification more reliable, you can enable the filelog.decompressFingerprint feature gate. With this enabled, the fingerprint is calculated from the decompressed content.

# docker-compose.yml
services:
  otelcol:
    command:
      [
        --config=/etc/otelcol-contrib/config.yaml,
        --feature-gates=filelog.decompressFingerprint,
      ]

One caution: if you turn this feature on in an existing setup, the fingerprints will change. That means compressed files that were already read may be ingested again.

Performance tuning for high-volume environments

The OTel collector filelog receiver's default settings are optimized for general use, but in production environments with hundreds of log files or very high throughput, you'll likely need to tune its performance.

By default, the receiver tries to read from every matched file at once. On a system producing thousands of files, this can hog the CPU and quickly hit file handle limits.

The max_concurrent_files setting puts a cap on how many files are read at the same time. The default is 1024, but lowering this can keep your system from getting overwhelmed.

Another key setting is poll_interval, which controls how often the receiver checks for new files and new log lines. The default is 200ms which means logs show up almost immediately, but CPU use goes up because the filesystem is scanned more often.

For less critical logs or resource-constrained environments, bumping this to 1s or even 5s can be a good trade-off as it'll reduce the polling overhead with only a negligible impact on observability for most use cases.

Finally, unusually large log entries are guarded against through the max_log_size setting. It defines the largest allowed log entry, so that anything bigger gets truncated. The default is 1MiB, which is a sensible default for most workloads.

# otelcol.yaml
receivers:
  filelog/k8s_pods:
    include: [/var/log/pods/*/*/*.log]
    max_concurrent_files: 200
    poll_interval: 1s
    max_log_size: 2MiB

Enforcing log file order

Most of the time, the order in which log files are ingested doesn't matter. But some systems produce logs as a series of sequential files where processing order is critical.

By default, the filelog receiver reads all matching files concurrently, which means you could end up processing them out of sequence. The ordering_criteria setting solves this by enforcing a strict order when reading files.

For example, given a set of log files with the following conventions:

batch-run-001.log
batch-run-002.log
batch-run-003.log

# otelcol.yaml
receivers:
  filelog/batch_logs:
    include: [/var/log/batch-runs/batch-run-*.log]
    start_at: beginning
    ordering_criteria:
      top_n: 1
      # Extract the sequence number from the filename
      regex: batch-run-(?P<seq_num>\d+)\.log
      # Sort files by the sequence number as a number, not a string
      sort_by:
        - regex_key: seq_num
          sort_type: numeric
          ascending: true

With this setup, the receiver will discover all files matching batch-run-*.log, extract the sequence number from each filename, and sort the files numerically in ascending order by that sequence.

The top_n property determines how many files will be tracked after applying the ordering criteria. With top_n: 1, only the first file (batch-run-001.log) will be tracked and ingested into the pipeline.

Filelog receiver tips and best practices

When troubleshooting the filelog receiver, a few issues come up again and again. Here's how to diagnose and fix them quickly:

Log files are not being watched

When the Collector starts watching a file for log entries, you'll see a log like this in its output:

2025-10-09T08:47:05.574Z        info    fileconsumer/file.go:261        Started watching file   {...}

If you don't see this message, or if you see the log below, it means that the receiver hasn't picked up any files yet:

2025-10-09T09:25:20.280Z        warn    fileconsumer/file.go:49 finding files   {..., "error": "no files match the configured criteria"}

Start by double-checking your include, exclude, and exclude_older_than settings to make sure your file patterns actually match the files you expect.

Next, verify that the Collector process has permission to access both the files and their parent directories. Missing directory-level permissions are one of the most common reasons files aren't discovered or watched.

Files are watched but no log lines are read

If you can see "Started watching file" messages but no logs are being collected, the most common cause is the start_at setting. By default, it's set to end, which tells the receiver to start reading only new lines appended after the Collector starts.

When you're testing with an existing file that isn't actively being written to, this means nothing will appear. To read the entire file from the start, set start_at to beginning:

# otelcol.yaml
receivers:
  filelog:
    start_at: beginning

This ensures the receiver processes all existing content the first time the file is discovered.

Regular expression doesn't match log lines

If your logs aren't being parsed correctly, the issue is usually with your regular expression. When this happens, the Collector often logs an error like:

2025-10-09T09:32:14.949Z        error   helper/transformer.go:154       Failed to process entry {"resource": {"service.instance.id": "f8ec2efd-16e9-44ad-9ed2-9f406e46719f", "service.name": "otelcol-contrib", "service.version": "0.136.0"}, "otelcol.component.id": "filelog", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "operator_id": "regex_parser", "operator_type": "regex_parser", "error": "regex pattern does not match", "action": "send", "entry.timestamp": "0001-01-01T00:00:00.000Z", "log.file.name": "batch-run-001.log"}

Before adjusting your Collector config, test the regex outside of it using a tool like Regex101. Make sure to select the Golang flavor so it behaves the same way as the Collector's regex engine.

If you're not seeing this error but your regex still isn't working, check whether the on_error parameter is set to one of the _quiet modes. Those values suppress operator errors unless the Collector log level is set to DEBUG.

Common causes of regex mismatches include invisible spaces or tabs, missing anchors (^ or $), incorrect escaping, or small format differences between your log and your pattern. Double-check these details before investigating further.

Logs are duplicated after restart

If you notice duplicate logs appearing after the Collector restarts, it usually means the receiver isn't remembering where it left off. To fix this, enable a storage extension so the filelog receiver can checkpoint its position in each file.

This allows the receiver to resume reading exactly where it stopped, preventing both data loss and duplication. Without it, the receiver will reread entire files from the start after every restart.

Final thoughts

The filelog receiver in OpenTelemetry is an essential bridge between traditional file-based logging (often with unstructured data) and the world of modern, structured observability.

By mastering its core concepts of discovery, parsing with operators, and checkpointing, you can build a reliable log ingestion pipeline for any service that writes its logs to a file.

Once you've transformed your raw text logs into well-structured OpenTelemetry data, the full observability ecosystem opens up. You can enrich, filter, and route them to any backend that speaks OTLP.

For a faster path from collecting telemetry to insight, consider using Dash0, an observability platform purpose-built for OpenTelemetry data. Try it out today with a free 14-day trial.

Leveling Up Your Python Logs with Structlog

Ayooluwa Isaiah — Thu, 27 Nov 2025 13:08:38 +0000

Python's standard logging module is capable, but shaping it into a system that produces structured, contextual, and queryable logs requires understanding a lot of concepts: hierarchical logging, formatters, filters, handlers, and configuration files. It can be done, but it often feels like you are building infrastructure instead of writing your application.

Structlog takes a different approach. Rather than wrestling with object hierarchies, you simply declare how each event should be processed and enriched. The result is logging that feels natural to write, while producing output that works just as well for humans skimming a console as it does for machines ingesting JSON into an observability platform.

This guide takes a practical look at using Structlog as the foundation for a production-grade logging system. We will cover configuration, contextual data, structured exception handling, and integration with tracing via OpenTelemetry. By the end, you'll have the patterns you need to turn your application from an opaque box into one that is transparent and easy to understand.

Let's begin!

Understanding the Structlog philosophy

The standard library's logging module is built around a small network of objects. You create a Logger, attach one or more Handler instances, give each handler a Formatter, and sometimes add Filters. A LogRecord is created and handed off to that graph. It works, and it is flexible, but it can be hard to follow.

Structlog takes a simpler path where each log event moves through a clear, linear chain of functions called processors. When your code calls something like logger.info("User logged in", user_id="usr_123"), structlog immediately builds a mutable dictionary for that event that looks like this:

{ "event": "User logged in", "user_id": "usr_123" }

That dictionary is then passed to each registered processor in order. A processor is just a function that gets three arguments: the logger, the method name, and the event dictionary. It can read the dictionary, add keys, remove keys, or tweak values.

The last processor is the renderer. Its job is to turn the final dictionary into a string and write it to your chosen destination, such as the console, a file, or a socket.

This declarative model is incredibly powerful because it provides a single source of truth. You can look at your list of processors and know exactly how a log entry is built, step by step. There is no hidden state or complex object interaction. It is a clean, predictable, and easily debuggable flow.

Examining the default configuration

Before diving into custom setups, it is helpful to see what Structlog does out of the box. The library ships with a default configuration that produces development-friendly logs without requiring you to write any setup code.

First, install the library if you haven't already:

pip install structlog

Now try the simplest possible logger:

import structlog

logger = structlog.get_logger()
logger.info(
    "User profile updated",
    user_id="usr_f4b7a1c2",
    request_id="req_9e8d5c3a-7b1f-4a8e-9c6d-0e2f1a3b4c5d",
    updated_fields=["email", "last_login"],
    duration_ms=54.3,
    status="success"
)

When you run this, you should see output along the lines of:

2025-09-05 18:13:33 [info     ] User profile updated           [__main__] duration_ms=54.3 request_id=req_9e8d5c3a-7b1f-4a8e-9c6d-0e2f1a3b4c5d status=success updated_fields=['email', 'last_login'] user_id=usr_f4b7a1c2

What you're seeing is a nicely formatted and colorized log line with a timestamp, the log level (info), and the message text, and finally the included key/value pairs.

Behind the scenes, Structlog is quietly applying a handful of processors that enrich and format each event before it gets written out. Here's the default configuration:

structlog.configure(
    processors=[
        structlog.contextvars.merge_contextvars,
        structlog.processors.add_log_level,
        structlog.processors.StackInfoRenderer(),
        structlog.dev.set_exc_info,
        structlog.processors.TimeStamper(fmt="%Y-%m-%d %H:%M:%S", utc=False),
        structlog.dev.ConsoleRenderer()
    ],
    wrapper_class=structlog.make_filtering_bound_logger(logging.NOTSET),
    context_class=dict,
    logger_factory=structlog.PrintLoggerFactory(),
    cache_logger_on_first_use=False
)

Here, the log event passes through the processors in sequential order. Each one adds a small piece of structure: merging in any contextual values you've set elsewhere, attaching the log level, making sure exceptions or stack traces are displayed neatly when they occur, and including a timestamp.

The very last step is the ConsoleRenderer(), which takes the fully enriched event dictionary and turns it into the formatted, colorized line you see in your terminal.

The other default behaviors of the Structlog logger are:

wrapper_class: This wrapper gives you a logger that can filter messages by log level. With NOTSET, nothing is filtered out, so every message goes through. In practice, you'd raise this to INFO or higher in production to reduce noise.
context_class: Structlog needs somewhere to store event data as it flows through the pipeline. By default, it uses a plain Python dictionary to keep things simple and predictable, but you could swap in something else (like OrderedDict) if you want ordered keys or a custom data structure.
logger_factory: This determines where your logs are sent, which is sys.stdout by default. You can switch it to the standard error with structlog.PrintLoggerFactory(sys.stderr).
cache_logger_on_first_use: By default, Structlog doesn't cache loggers. That means every call to get_logger() creates a new one, which ensures that if you change the configuration at runtime, the new settings are applied immediately. If performance is critical, you can enable caching for a small speed boost.

Put together, these defaults make Structlog easy to experiment with: nothing gets filtered, logs print straight to your terminal, and you can reconfigure on the fly without restarting your process. It's a developer-friendly setup that you'll want to tighten up before going to production.

The production configuration: machine-readable JSON

In a production environment, the requirements are different. Logs are not primarily for humans to read in real-time; they are for machines to ingest, parse, index, and query. The industry standard for this is JSON.

Our production configuration will be similar to the default, but with a few changes:

def configure_structlog(log_level=logging.INFO):
    structlog.configure(
        processors=[
            structlog.contextvars.merge_contextvars,
            structlog.processors.add_log_level,
            structlog.processors.StackInfoRenderer(),
            structlog.dev.set_exc_info,
            structlog.processors.dict_tracebacks,
            structlog.processors.TimeStamper(fmt="iso"),
            structlog.processors.JSONRenderer() # must be the last one
        ],
        wrapper_class=structlog.make_filtering_bound_logger(log_level),
        context_class=dict,
        logger_factory=structlog.PrintLoggerFactory(),
        cache_logger_on_first_use=True
    )

The changes here include:

Renderer (ConsoleRenderer → JSONRenderer): Instead of a pretty, colorized line for humans, each event becomes a single JSON object so that log shippers and observability platforms can ingest it without guessing at formats or using brittle regular expressions.
Timestamp (fmt="iso"): Timestamps use ISO 8601 format in UTC, which avoids timezone confusion and preserves correct lexicographical ordering, especially across regions.
Dict tracebacks (dict_tracebacks): Exceptions are serialized into structured dictionaries instead of raw text. This makes stack traces machine-readable, so that observability tools can display them cleanly, and you can query or filter logs by exception type or message.
Configurable log level: The log level is now passed in as an argument, allowing you to control log verbosity in production without changing code, typically by reading an environment variable.
Caching: In production, you rarely hot-reload logging configuration, so caching gives a small performance boost by avoiding repeated wrapper setup.

Now, you can dynamically choose your configuration at startup:

import os
import structlog
import logging

# ... [logging configuration]

if os.environ.get("APP_ENV") == "production":
    log_level = os.environ.get("LOG_LEVEL", logging.INFO)
    configure_structlog(log_level)

logger = structlog.get_logger()
# ... rest of your application logic

This pattern allows you to retain Structlog's development-friendly defaults, while switching to a production-ready JSON configuration automatically when the environment demands it.

Note that while Structlog operates independently, it adopts the same level names and numeric values as the standard library. For convenience and clarity, we use the constants from the logging module (like logging.INFO) to set these levels.

Assuming you set APP_ENV=production in your environment, you'll see the following JSON output:

{
  "user_id": "usr_f4b7a1c2",
  "request_id": "req_9e8d5c3a-7b1f-4a8e-9c6d-0e2f1a3b4c5d",
  "updated_fields": ["email", "last_login"],
  "duration_ms": 54.3,
  "status": "success",
  "event": "User profile updated",
  "level": "info",
  "timestamp": "2025-09-06T07:40:44.956022Z"
}

The log message is placed in an event key, but you can rename it to msg by using the EventRenamer() processor as follows:

processors=[
    # [...]
    structlog.processors.EventRenamer("msg"),
    structlog.processors.JSONRenderer()
]

The event key will be renamed to msg accordingly:

{
  [...]
  "timestamp": "2025-09-06T07:46:58.238599Z",
  "msg": "User profile updated"
}

The examples in the remainder of this article will assume that you're using the production configuration.

How log levels work in Structlog

Structlog keeps the same log levels you may already know from Python's standard logging module:

Level	Numeric Value	Description
`NOTSET`	0	Special: disables log-level filtering
`DEBUG`	10	Detailed diagnostic information
`INFO`	20	Normal application events
`WARNING`	30	Potential problems
`ERROR`	40	Failed operations
`CRITICAL`	50	Severe failures

Each level besides NOTSET has a corresponding method on the logger:

logger.debug("a debug message")
logger.info("an info message")
logger.warning("a warning message")
logger.error("a error message")
logger.critical("a critical message")

If you're working in an async context, Structlog also provides async variants which are prefixed with a:

import structlog
import asyncio

logger = structlog.get_logger()

async def f():
    await logger.ainfo("async info message")

asyncio.run(f())

As you've already seen, the level threshold is controlled by the wrapper_class argument to structlog.configure():

structlog.configure(
    wrapper_class=structlog.make_filtering_bound_logger(<log_level>),
)

The argument to make_filtering_bound_logger() could be a simple string (like "INFO") or the constants on the logging module (such as logging.INFO). Level-based filtering is done early before the event dictionary is created to avoid doing unnecessary work for a message that will ultimately be discarded.

Structlog also makes the log level explicit inside the event dictionary itself. This happens thanks to the add_log_level processor, which is included in the default configuration.

Downstream processors (like the renderer) then use that field to decide how the log line should appear---whether that's a colorized console message in development or a structured JSON object in production.

Filtering and dropping events

In some cases, log level filtering isn't enough. You may want to drop or modify logs based on their content---for example, to exclude noisy health checks or to mask sensitive fields. You can do this with a custom processor:

def filter_logs(logger, method_name, event_dict):
    if event_dict.get("path") == "/health":
        raise structlog.DropEvent
    return event_dict

structlog.configure(
    processors=[
        filter_logs,
        # [...]
    ],
)

If a processor raises structlog.DropEvent, the event is discarded and no log line is emitted.

Filtering by call site information

Sometimes you don't just want to filter logs by level or custom fields; you want to filter them based on where they came from. Structlog makes this possible with the CallsiteParameterAdder, which can enrich your event dictionary with details like the module name, function name, line number, or thread ID. Once those fields are available, you can write a processor that decides which events to keep.

Let's say you have a simple application with two operations: processing an order and canceling an order:

logger = structlog.get_logger()

def process_order(order_id):
    logger.info("Order processed successfully", order_id=order_id)

def cancel_order(order_id):
    logger.warning("Order canceled", order_id=order_id)

process_order("ord_123")
cancel_order("ord_456")

This produces:

{"order_id": "ord_123", "level": "info", "timestamp": "2025-09-06T13:59:03.397454Z", "msg": "Order processed successfully", "func_name": "process_order"}
{"order_id": "ord_456", "level": "warning", "timestamp": "2025-09-06T13:59:03.397618Z", "msg": "Order canceled", "func_name": "cancel_order"}

Now, suppose you only care about logs from process_order and want to ignore everything else. You can add a custom processor that drops events from the unwanted function:

def filter_out_cancellations(_, __, event_dict):
    if event_dict.get("func_name") == "cancel_order":
        raise structlog.DropEvent
    return event_dict

structlog.configure(
    processors=[
        structlog.processors.CallsiteParameterAdder(
            [structlog.processors.CallsiteParameter.FUNC_NAME]
        ),
        filter_out_cancellations # must be placed after CallsiteParameterAdder
    ]
)

With this configuration, calling both functions again yields:

{
  "order_id": "ord_123",
  "level": "info",
  "timestamp": "2025-09-06T13:59:03.397454Z",
  "msg": "Order processed successfully",
  "func_name": "process_order"
}

The cancel_order log entry has now been filtered out.

The reason this setup works is that CallsiteParameterAdder adds details about where the log call was made, such as the function name. Once that information is present in the event dictionary, the custom filter_out_cancellations processor can examine it and decide what to do. If the function name matches cancel_order, it raises DropEvent, which tells Structlog to discard the log entirely.

Because processors are executed in order, the event first gains the extra metadata, then it is evaluated by the filter, and finally the surviving events are handed off to the renderer. The result is that only logs from process_order appear in the output, while logs from cancel_order are silently filtered out.

Writing logs to files

The 12-Factor App methodology recommends writing logs to standard output and letting the platform handle collection, and that's still the best approach in containerized and cloud environments. However, some deployments do require logs to be written directly to files.

In such cases, you can configure the PrintLoggerFactory as follows so logs are sent to a file instead of stdout:

import structlog
from pathlib import Path

structlog.configure(
    processors=[...],
    logger_factory=structlog.PrintLoggerFactory(
        file=Path("app").with_suffix(".log").open("wt")
    ),
)

You can also use the WriteLoggerFactory which the documentation claims is "a little faster" than PrintLoggerFactory at the cost of some versatility.

Structlog itself doesn't handle rotation or retention, but leaves such tasks to a dedicated system utility like Logrotate.

Mastering contextual logging

The single most important practice that separates log records from a stream of messages into a true observability signal is context.

In Structlog, every logging call lets you attach structured key--value pairs alongside your message. These fields travel with the log event through the processor pipeline and end up in the final output:

logger.info(
    "User profile updated",
    user_id="usr_f4b7a1c2",
    request_id="req_9e8d5c3a",
    status="success",
    duration_ms=54.3
)

Instead of just a sentence like "User profile updated" you now have rich, machine-readable details: which user was affected, which request triggered the change, whether it succeeded, and how long it took.

You can also bind context to a logger so that it's included automatically in every message from that logger. For instance:

# bind() returns a copy of `logger` with user_id added to its context
logger = logger.bind(user_id="usr_f4b7a1c2")
logger.info("Fetching user profile")
logger.info("Profile fetched successfully")

Both log lines will now include the user_id field without you having to repeat it each time:

{"user_id": "usr_f4b7a1c2", "level": "info", "timestamp": "2025-09-06T09:00:25.822429Z", "msg": "Fetching user profile"}
{"user_id": "usr_f4b7a1c2", "level": "info", "timestamp": "2025-09-06T09:00:25.822570Z", "msg": "Profile fetched successfully"}

If later on you decide to remove the bound fields from the logger's context, you can use unbind() and try_unbind():

logger = logger.unbind("user_id") # will throw an error if the key doesn't exist
logger = logger.try_unbind("user_id") # missing keys are ignored

Reliable context propagation in web apps

While bind() is useful, it has limitations in highly concurrent environments like web applications. You don't want to pass a request-specific logger instance down through every function call as it quickly becomes clumsy.

A much more powerful and elegant solution is to use context variables through structlog.contextvars. This takes advantage of Python's contextvars module to store context that is scoped to the current thread or async task.

Each request (or background job) gets its own isolated context, so you never have to worry about data leaking between concurrent executions.

That's why our production configuration includes the structlog.contextvars.merge_contextvars processor for pulling the context into each log event automatically.

All you need to do is bind values at the beginning of a request or task, and those values will show up in every log line until the context is cleared.

Here's an example of how you might set this up in a FastAPI middleware:

import uuid
from fastapi import FastAPI, Request, HTTPException
import structlog
import time
import logging

# ...your structlog configuration

logger = structlog.get_logger()

app = FastAPI()

@app.middleware("http")
async def context_middleware(request: Request, call_next):
    start_time = time.monotonic()

    # This is the request ID that will be attached to all logs for this request
    request_id = request.headers.get("X-Request-ID", str(uuid.uuid4()))
    client_ip = request.headers.get("X-Forwarded-For", request.client.host)
    user_agent = request.headers.get("user-agent", "-")

    # Clear any existing context from previous requests
    structlog.contextvars.clear_contextvars()
    # All logs produced in this request will share the same request_id
    structlog.contextvars.bind_contextvars(
        request_id=request_id,
    )

    request_logger = logger.bind(
        method=request.method,
        path=request.url.path,
        client_ip=client_ip,
        user_agent=user_agent,
    )

    request_logger.info("Incoming %s request to %s", request.method, request.url.path)

    response = await call_next(request)

    duration_ms = (time.monotonic() - start_time) * 1000

    log_level = logging.INFO

    if response.status_code >= 500:
        log_level = logging.ERROR
    elif response.status_code >= 400:
        log_level = logging.WARNING

    request_logger.log(
        log_level,
        "%s %s completed with status %s",
        request.method,
        request.url.path,
        response.status_code,
        status=response.status_code,
        duration=duration_ms,
    )


    return response

@app.get("/users/{user_id}")
async def get_user_profile(user_id: str):
    # You can even add more context, scoped just to this function
    # and any downstream functions
    structlog.contextvars.bind_contextvars(user_id=user_id)

    logger.info("User profile requested.")

    if user_id == "error":
        raise HTTPException(status_code=404, detail="Item not found")

    logger.info("Successfully retrieved user profile.")
    return {"user": user_id, "status": "ok"}

if __name__ == "__main__":
    import uvicorn

    uvicorn.run(
        app,
        host="0.0.0.0",
        port=8000,
        reload=False,
        access_log=False,
    )

With this setup, every request to your API gets its own request ID bound to the logging context, so that it's automatically included in all log messages during that request, without you having to pass a logger or request_id around manually.

{"method": "GET", "path": "/users/12", "client_ip": "127.0.0.1", "user_agent": "curl/8.7.1", "level": "info", "request_id": "510ec4b6-f27f-4380-9082-487a3193094e", "timestamp": "2025-09-06T10:33:03.077963Z", "msg": "Incoming GET request to /users/12"}
{"level": "info", "request_id": "510ec4b6-f27f-4380-9082-487a3193094e", "user_id": "12", "timestamp": "2025-09-06T10:33:03.078208Z", "msg": "User profile requested."}
[...]

Within the scope of a function you can continue to use bind() if you intend to add temporary fields that are only relevant to a narrow slice of work (see the request_logger for access logging), or call structlog.contextvars.bind_contextvars() again to add new fields and pass them on to loggers in downstream functions.

Adding global context

If you need some variables to appear in every single log record regardless of how the logger is obtained, you can add them with a processor that runs for each event.

You only need to capture the values at startup, then merge them into the event dictionary:

import logging
import os
import os as _os
import socket
import structlog

APP_VERSION = "2.4.1"

def add_global_fields_factory():
    service = os.getenv("SERVICE_NAME", "user-service")
    env = os.getenv("APP_ENV", "development")
    region = os.getenv("REGION", "local")
    host = socket.gethostname()
    pid = _os.getpid()
    version = APP_VERSION

    def add_global_fields(logger, method_name, event_dict):
        event_dict.setdefault("service", service)
        event_dict.setdefault("env", env)
        event_dict.setdefault("region", region)
        event_dict.setdefault("version", version)
        event_dict.setdefault("host", host)
        event_dict.setdefault("pid", pid)
        return event_dict

    return add_global_fields

structlog.configure(
    processors=[
        add_global_fields_factory(), # add this as the first processor
        # [...]
    ]
)

Each record from the service will now contain the global fields:

{
  "service": "user-service",
  "env": "development",
  "region": "us-east-1",
  "version": "2.4.1",
  "host": "MacBook-Pro.local",
  "pid": 82904,
  "level": "info",
  "request_id": "4af0acc1-1064-4470-a538-bb9862cd2154",
  "user_id": "12",
  "timestamp": "2025-09-06T10:18:14.194947Z",
  "msg": "Successfully retrieved user profile."
}

Capturing Python errors and exceptions

When an error occurs in production, your logs are your first and often best debugging tool. A plain traceback string is helpful, but a structured exception record is far more powerful. This is exactly what the dict_tracebacks processor gives you.

The key to this is using logger.exception(). While you can log errors with logger.error(), using logger.exception() inside an except block is the preferred pattern. It automatically captures the active exception and passes it through the pipeline:

structlog.configure(
    processors=[
        structlog.processors.dict_tracebacks # ensure dict_tracebacks is configured
        # [...]
    ],
)

try:
    1 / 0
except Exception:
    logger.exception("Dividing by zero")

With the dict_tracebacks processor enabled, the resulting JSON log contains a fully structured representation of the exception. Here's a simplified example:

{
  "level": "error",
  "msg": "Dividing by zero",
  "timestamp": "2025-09-06T12:28:16.625474Z",
  "exception": [
    {
      "exc_type": "ZeroDivisionError",
      "exc_value": "division by zero",
      "frames": [
        {
          "filename": "main.py",
          "lineno": 37,
          "name": "<module>",
          "locals": {
            "logger": "<BoundLoggerLazyProxy ...>"
          }
        }
      ]
    }
  ]
}

Instead of a traceback string, you now have a structured object that exposes the exception type, value, and even stack frames. This structure unlocks powerful new workflows in your log aggregation system. For example, you can:

Query for all logs where exception.exc_type equals ZeroDivisionError,
Count how many errors originated in a function by filtering on exception.frames[0].name,
Trigger alerts if exception.exc_value contains a specific string.

With structured tracebacks, your logs become more than just text. They become queryable data that dramatically reduces the time it takes to detect, diagnose, and fix production issues.

In development environments, you can install the rich library to render a colorful traceback in the terminal:

Integrating Structlog with OpenTelemetry

Structlog does not automatically add trace or span identifiers to your logs. To correlate logs with traces, you attach those fields yourself with a small processor that reads the current OpenTelemetry span and injects its IDs into the event dictionary.

Once in place, every log written inside an active span will carry trace_id and span_id, which makes it possible to see your spans and logs in the same context.

Below is a compact setup that wires Structlog to OpenTelemetry and adds the two IDs on every event:

import os
import structlog
import logging
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider

trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)
# [... rest of your tracing configuration]

def add_open_telemetry_spans(_, __, event_dict):
    span = trace.get_current_span()
    if not span or not span.is_recording():
        return event_dict

    ctx = span.get_span_context()
    parent = getattr(span, "parent", None)

    event_dict["span_id"] = format(ctx.span_id, "016x")
    event_dict["trace_id"] = format(ctx.trace_id, "032x")

    return event_dict

structlog.configure(
    processors=[
        # [...]
        add_open_telemetry_spans,
        structlog.processors.JSONRenderer() # must be the last one
    ]
)

logger = structlog.get_logger()

def process_order(order_id):
    with tracer.start_as_current_span("process order") as span:
        logger.info("Order processed successfully", order_id=order_id)

process_order("ord_123")

When the logger.info() call runs inside an active span, your JSON log will now include the OpenTelemetry identifiers:

{
  "order_id": "ord_123",
  "level": "info",
  "timestamp": "2025-09-07T07:13:43.906388Z",
  "msg": "Order processed successfully",
  "span_id": "da8405273d89b065",
  "trace_id": "442d81cb25de382054575e33c1a659df"
}

What's left is bringing your logs into an OpenTelemetry-native platform like Dash0 where they can be filtered and correlated with other signals like metrics and traces to give you a complete picture of your system's health.

At the time of writing, there is no way to export native OTLP logs directly from Structlog, so you have two options.

First, keep Structlog writing JSON to standard output or a file and let the OpenTelemetry Collector convert it to OTLP log schema, before forwarding to your backend. This keeps the application simple and pushes protocol concerns to the infrastructure.

The second option requires that you bridge Structlog to the standard logging ecosystem and attach an OTLP-capable handler. You'll need to configure structlog.stdlib.LoggerFactory() and a ProcessorFormatter, then attach a handler that exports to an endpoint speaking OTLP (usually the OTel Collector).

Final thoughts

Structlog turns Python logging into a stream of structured events rather than plain text. With its processor pipeline, you can enrich logs with context, filter noise, and render output in formats that suit both humans and machines.

We began with simple console logs, then moved to production configurations with JSON rendering, timestamps, and structured exceptions. We explored how bind() and contextvars add valuable context, how callsite parameters provide control, and how integration with OpenTelemetry connects logs to traces.

The takeaway is clear: logs are data. Treating them as structured signals with Structlog makes debugging, monitoring, and operating modern applications far more effective.

Thanks for reading!

Mastering Docker Logs: A Comprehensive Tutorial

Ayooluwa Isaiah — Fri, 14 Nov 2025 13:15:43 +0000

You've just deployed a new feature. It's not on fire, but it's not quite right either. An API response is missing a field, and performance seems a bit off. Where do you begin to unravel the mystery? You start with the logs.

In a containerized environment, however, logging isn't always straightforward. Logs are ephemeral, dispersed across multiple containers, and can grow unmanageable without the right strategy.

This guide covers everything you need to know about Docker logs. We'll start with the simplest commands to view logs in real-time and progress to designing a robust, production-grade logging strategy for your entire containerized infrastructure.

Let's get started!

Quick start: the `docker logs` command reference

For when you need answers now, here are the most common commands you'll reach for often:

Action	Command
View all logs for a container	`docker logs <container>`
Follow logs in real-time (tail)	`docker logs -f <container>`
Tail the last 100 lines	`docker logs --tail 100 <container>`
View logs from the last 15 minutes	`docker logs --since 15m <container>`
View logs for a Docker Compose service	`docker compose logs <service>`
Follow logs for all Compose services	`docker compose logs -f`
Remove the service prefix in Docker Compose logs	`docker compose logs --no-log-prefix <service>`

Understanding how Docker logging works

Docker is designed to capture the standard output (stdout) and standard error (stderr) streams from the main process running inside a container.

This means that if you are containerizing your own services, you should ensure that they're writing their logs to stdout or stderr so that Docker's built-in logging system can capture them.

A logging driver acts as the backend for these logs. It receives the log streams from the container and determines whether to store them in a file or forward them to an endpoint.

Where Docker stores container logs

By default, Docker uses the json-file logging driver to write the captured container logs to a file on the host machine.

Here's the typical Docker logs location for a container:

/var/lib/docker/containers/<container-id>/<container-id>-json.log

You can find the log file path for a specific container with this command:

docker inspect -f '{{.LogPath}}' <container_name_or_id>

This outputs:

/var/lib/docker/containers/612646b55e41d73a3f1a24afa736ef173981ed753506097d1a888e7b9cb7d6ac/612646b55e41d73a3f1a24afa736ef173981ed753506097d1a888e7b9cb7d6ac-json.log

In most cases, you won't need to interact with these log files directly as it's what docker logs reads from behind the scenes.

If you're unsure of what logging driver a container uses, you can confirm with:

docker inspect -f '{{.HostConfig.LogConfig.Type}}' <container_name_or_id>

Viewing container logs with the `docker logs` command

The docker logs command is the primary way to inspect the logs of a running or stopped container. It is a shorthand for the full docker container logs command and can be used interchangeably.

Here's the built-in usage reference for quick context:

Usage:  docker logs [OPTIONS] CONTAINER

Fetch the logs of a container

Aliases:
  docker container logs, docker logs

Options:
      --details        Show extra details provided to logs
  -f, --follow         Follow log output
      --since string   Show logs since timestamp (e.g. "2013-01-02T13:23:37Z") or relative (e.g. "42m" for 42 minutes)
  -n, --tail string    Number of lines to show from the end of the logs (default "all")
  -t, --timestamps     Show timestamps
      --until string   Show logs before a timestamp (e.g. "2013-01-02T13:23:37Z") or relative (e.g. "42m" for 42 minutes)

To view all available logs for a container, simply pass its name or ID:

docker logs <container_name_or_id>

This dumps the entire log history of the specified container to your terminal, which is probably not what you're after.

For a container that's been running for a while, or one that's particularly noisy, this can mean scrolling through thousands of lines of output.

To isolate the specific information you need, you can use Docker's built-in filtering flags to narrow the output by time or by the number of lines.

Let's explore the most useful options next. Note that all options must come before the container name or ID:

docker logs [<options>] <container_name_or_id>

Note: If docker logs isn't showing anything, you may be hitting one of the common pitfalls. See the troubleshooting section to learn how to fix it.

Filtering logs by time (`--since` and `--until`)

To constrain docker logs output to a specific time window, you can use a combination of the following options:

--since: Shows logs generated after a specified point in time.
--until: Shows logs generated before a specified point in time.

With either flag, you can provide a relative time (like 10m for 10 minutes, 3h for 3 hours) or an absolute timestamp (such as 2025-06-13T10:30:00).

# Show logs from the last 30 minutes
docker logs --since 30m <container_name_or_id>

# Show logs from this morning, before 10 AM
docker logs --until 2025-06-13T10:00:00 <container_name_or_id>

You can also combine the two:

docker logs --since 2025-06-13T18:00:00 --until 2025-06-13T18:15:00 <container_name_or_id>

Following and tailing Docker container logs

While filtering logs by time helps you understand historical events, the most common task while troubleshooting is to see what's happening right now.

This means continuously monitoring and displaying the end of a log file in real-time as new entries are added.

To follow Docker container logs, use the -f or --follow flag:

docker logs -f <container_name_or_id>

However, this will print the container's entire log history before it starts streaming new entries, which isn't ideal.

The most effective pattern is to combine --follow with --tail (or its shorthand -n). This gives you the best of both worlds: a small amount of recent history for context, followed by the live stream:

docker logs -f --tail 100 <container_name_or_id>

Here, only the last 100 lines are displayed for context, and then new log messages stream in real time. When you want to stop streaming, press Ctrl+C.

Filtering Docker logs with `grep`

The docker logs command doesn't have a built-in grepping feature, but you can easily pipe its output to standard shell utilities like grep to quickly search for a specific string:

docker logs <container_name_or_id> | grep -i "ERROR"

This helps you filter out noise and return only the log lines that match your search term.

Saving Docker logs to a file

If you need to store a subset of your logs for later analysis, you can redirect the output of docker logs to a file:

docker logs <container_name_or_id> > app.log       # overwrite
docker logs <container_name_or_id> >> app.log      # append

You can also combine this with filters:

docker logs --since 10m <container_name_or_id> > recent.log

This is the simplest way to capture Docker logs to a file without changing any logging configuration.

Managing logs in Docker Compose

Most Docker projects use Docker Compose, and managing logs there is just as straightforward. The main difference is that you use docker compose logs rather than docker logs.

The usage syntax is:

docker compose logs [options] [service...]

Where [service...] is an optional list of service names. Since a single service may be running across multiple containers, Docker Compose automatically aggregates the logs from all containers that belong to that service.

Let's look at a few common usage patterns.

Viewing logs for a single service

To see logs from just one service defined in your Compose file, specify the service name:

docker compose logs image-provider

You can also specify multiple service names:

docker compose logs image-provider shipping otel-collector

Docker Compose will color-code the output by service, making it easy to follow:

For ease of copying and pasting log lines, you'll want to include the --no-log-prefix flag:

docker compose logs --no-log-prefix <services>

Viewing logs for all services

To see an interleaved stream of logs from all services in your stack, run the command without a service name:

docker compose logs

Tailing and filtering

All the flags you learned for docker logs for tailing and filtering work with docker compose logs too:

docker compose logs --follow --tail 10 image-provider cart

docker compose logs --since '10m' db

Inspecting Docker container logs with a GUI

If you would rather inspect container logs visually, using a Docker log viewer can make it much easier to browse, filter, and search container logs without relying on the command line.

1. Docker Desktop logs

The built-in dashboard in Docker Desktop has a Logs tab for any running container. It provides a simple, real-time view with basic search functionality.

2. Dozzle

Dozzle is a lightweight, web-based log viewer with a slick interface. It's incredibly easy to run as a Docker container itself:

docker run -d --name dozzle \
    -p 8888:8080 \
    --volume /var/run/docker.sock:/var/run/docker.sock \
    amir20/dozzle:latest

Navigate to http://localhost:8888 in your browser to get a real-time view of all your container logs.

Choosing a Docker logging driver

While json-file is the default, Docker supports a variety of other logging drivers to suit different needs:

none: Disables logging entirely which is useful when logs are unnecessary or handled externally.
local: Recommended for most use cases. It offers better performance and more efficient disk usage than json-file.
syslog: Sends logs to the system's syslog daemon.
journald: Write log output to the journald logging system.
fluentd, gelf, awslogs, gcplogs, etc.: Forward logs to external logging services or cloud platforms for centralized aggregation and analysis.

Setting a global logging driver

To set a global logging driver for all Docker containers, you must edit the Docker daemon configuration file at /etc/docker/daemon.json. If the file doesn't exist, create it first:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "4",
    "compress": "true"
  }
}

If you're using Docker Desktop, you can edit this file by opening the app's Settings and selecting Docker Engine from the sidebar:

The json-file driver's most significant drawback is that it does not rotate logs by default. Over time, these log files will grow indefinitely, which can consume all available disk space and crash your server.

This configuration addresses this by telling Docker to:

Rotate log files when they reach 50MB (max-size).
Keep a maximum of four old log files (max-file).
Compress the rotated log files to save space (compress).

An alternative approach is using the local driver since it uses a more compact file format and includes log rotation out of the box:

{
  "log-driver": "local"
}

In production environments, where container logs must be shipped to an external observability platform, you can choose from other available logging drivers.

For example, this section below demonstrates how to bring Docker logs into an OpenTelemetry pipeline through the fluentd driver.

Once you've edited your daemon.json file, you must restart the Docker daemon for the changes to take effect for newly created containers. Existing containers need to be recreated to adopt the updated configuration.

sudo systemctl restart docker

Overriding the logging driver per container

You can override the global logging driver for individual containers when launching them. This is useful for containers that need different retention or delivery behavior:

docker run \
  --log-driver=local \
  --log-opt max-size=50m \
  --log-opt max-file=4 \
  <image_name>

If you're using Docker Compose, you can configure the logging driver in your docker-compose.yml file:

# docker-compose.yml
<service_name>:
  image: <image_name>
  logging:
    driver: "local"
    options:
      max-file: "4"
      max-size: "50m"
      compress: "true"

To avoid repeating the same configuration across multiple services, define a YAML anchor and reuse it like this:

# docker-compose.yml
x-default-logging: &logging
  driver: "local"
  options:
    max-size: "50m"
    max-file: "4"

services:
  <service_a>:
    logging: *logging

  <service_b>:
    logging: *logging

Understanding Docker's log delivery mode

When your application generates a log, it faces a fundamental choice: should it pause to ensure the log is safely delivered, or should it hand the log off quickly and continue its work?

This is the core trade-off managed by Docker's log delivery mode, a crucial setting that lets you tune your logging for either maximum reliability or maximum performance.

Docker supports two modes for delivering logs from your container to the configured logging driver.

1. Blocking mode

In the default blocking mode, log delivery is synchronous. When your application emits a log, it must wait for the Docker logging driver to process and accept that message before it can continue executing.

This approach is best for scenarios where every log message is critical and you are using a fast, local logging driver like local or json-file.

With slower drivers (those that send logs over a network), blocking mode can introduce significant latency and even stall your application if the remote logging service is slow or unreachable.

2. Non-blocking mode

As an alternative, you can configure a non-blocking delivery mode. In this mode, log delivery is asynchronous. When your application emits a log, the message is immediately placed in an in-memory buffer, and your application continues running without any delay. The logs are then sent to the driver from this buffer in the background.

The trade-off for this mode is a risk of losing logs. If the in-memory buffer fills up faster than the driver can process logs, new incoming messages will be dropped.

To mitigate the risk of losing logs in non-blocking mode, you can increase the size of the in-memory buffer from its 1MB default:

{
  "log-driver": "awslogs",
  "log-opts": {
    "mode": "non-blocking",
    "max-buffer-size": "50m"
  }
}

Centralizing Docker logs with OpenTelemetry

While local tools work well in development environments, production systems require a unified logging pipeline that ensures container logs are captured and retained, even after the container is long gone.

By consolidating your Docker logs in an observability platform like Dash0, you'll gain the ability to perform complex searches across your entire infrastructure, build real-time dashboards to visualize trends, and correlate logs with other telemetry signals like metrics or traces.

One of the most effective ways to ship Docker logs from each host to an observability service is through the OpenTelemetry Collector which supports a variety of log ingestion methods.

You may be tempted to use the filelog receiver to read container log files directly, but this is rarely ideal for Docker environments.

A more effective approach is to set up fluentd as the Docker logging driver for your services. This lets Docker stream logs to a Fluentd endpoint without relying on file scraping.

Here's the configuration you need in your daemon.json:

{
  "log-driver": "fluentd",
  "log-opts": {
    "fluentd-address": "localhost:8006",
    "tag": "opentelemetry-demo"
  }
}

Or when setting up the container from the command line:

docker run -d --name <container_name> --log-driver=fluentd --log-opt fluentd-address=localhost:8006 <image>

Or in your Docker Compose file:

# docker-compose.yml
services:
  <service>:
    image: <image>
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:8006
        tag: nginx.myapp

You can then configure the fluentforward receiver in your OpenTelemetry collector configuration to set up a server that's listening at the fluentd-address specified above:

# otelcol.yaml
receivers:
  fluentforward:
    endpoint: 0.0.0.0:8006

processors:
  batch:
  resourcedetection/system:
    detectors: [system]
    system:
      hostname_sources: [os]

exporters:
  otlphttp/dash0:
    endpoint: <your_dash0_endpoint>
    headers:
      Authorization: Bearer <your_dash0_token>
      Dash0-Dataset: <your_dash0_dataset>

service:
  pipelines:
    logs:
      receivers: [fluentforward]
      processors: [batch, resourcedetection/system]
      exporters: [otlphttp/dash0]

Once you replace the Dash0 placeholders with your actual account values, you can run the OpenTelemetry Collector as a sidecar:

docker run \
  -v $(pwd)/otelcol.yaml:/etc/otelcol-contrib/config.yaml \
  otel/opentelemetry-collector-contrib:latest

Then you'll start seeing your logs in the Dash0 interface.

Troubleshooting common issues with Docker container logs

Docker logging is generally straightforward, but a few recurring issues can still cause confusion. Here's how to recognize them and resolve them quickly.

1. `docker logs` shows no output

What's happening: Your application likely isn't writing to stdout or stderr. It might be logging directly to a file inside the container instead. Since Docker's logging drivers only capture standard output streams, it won't pick up logs written to internal files.

How to fix it: Ideally, update your application's logging configuration to write directly to stdout or stderr. If modifying the application isn't feasible, you can redirect file-based logs by creating symbolic links to the appropriate output streams in your Dockerfile.

# Example for an Nginx image
RUN ln -sf /dev/stdout /var/log/nginx/access.log && \
    ln -sf /dev/stderr /var/log/nginx/error.log

This ensures that even file-based logs are routed through Docker's logging mechanism.

2. Logging driver does not support reading

Error response from daemon: configured logging driver does not support reading

What's happening: Remote logging drivers such as awslogs, splunk, or gelf forward logs directly to an external system without storing anything locally. Normally, Docker caches the logs using its dual logging functionality. However, if this feature is disabled for the container, the docker logs command can't retrieve any output.

How to fix it: You need to ensure cache-disabled is false in the logging options. This tells Docker to send logs to the remote driver and keep a local copy for docker logs to use.

{
  "log-driver": "awslogs",
  "log-opts": {
    "cache-disabled": "false"
  }
}

Some best practices for Docker container logging

Effective logging in a containerized environment is about more than running docker logs. Following the guidelines below will help you build a logging setup that holds up in real-world Docker deployments.

1. Write application logs to stdout and stderr

Docker only captures what your application writes to stdout and stderr. Avoid writing logs directly to files inside the container unless you redirect them to these streams.

2. Use non-blocking mode for network-based logging drivers

Drivers that send logs over the network (such as fluentd, gelf, or awslogs) can back-pressure the application in blocking mode. It's usually better to enable non-blocking mode and tune max-buffer-size to avoid losing logs during spikes.

3. Include container metadata in logs

Metadata such as container name, ID, service name, and image version are crucial for troubleshooting containerized workloads in production. In an OpenTelemetry pipeline, such metadata belongs in the resource attributes, so that it travels with every log record.

4. Use the `local` or `json-file` driver with log rotation

If you rely on host-level log storage (via local or json-file), always enable rotation through the max-file and max-size options. Unrotated log files are a common cause of disk pressure on production nodes.

Final thoughts

You've now journeyed from the basic docker logs command to understanding the critical importance of logging drivers, log rotation, and centralized logging strategies.

By mastering these tools and concepts, you're no longer just guessing when things go wrong. You have the visibility you need to build, debug, and run resilient, production-ready applications.

Whenever possible, structure your application's logs as JSON. A simple text line is hard to parse, but a JSON object with fields like level, timestamp, and message is instantly machine-readable, making your logs infinitely more powerful in any observability platform.

Thanks for reading!

From Zero to Pipeline: An OpenTelemetry Collector Guide

Ayooluwa Isaiah — Tue, 11 Nov 2025 15:23:38 +0000

Your services likely emit a constant steam of telemetry data, but how does it actually travel from your applications and infrastructure to the observability backend where it's stored and analyzed?

For many, the answer is a chaotic web of vendor-specific agents, direct-to-backend SDK configurations, and disparate data shippers. This setup is brittle, expensive, hard to manage, and locks you into a single vendor's ecosystem.

There is a better way.

Instead of managing a maze of point-to-point integrations, we're going to build a telemetry pipeline: a centralized, vendor-neutral system that gives you complete control to collect, enrich, and route your observability data.

At the heart of this system is the OpenTelemetry Collector. It is a standalone service that acts as a universal receiver, a powerful processing engine, and a flexible dispatcher for telemetry data.

In this article, we'll build a telemetry pipeline from the ground up. You'll move from configuring basic data ingestion and exporting to discovering several processing techniques and designing complex data flows that help turn raw telemetry into actionable insights.

Let's get started!

The simplest possible pipeline

Every data pipeline needs an entry point and an exit. We'll start by building the most basic version of an OpenTelemetry pipeline imaginable. The goal is to receive telemetry data and print it directly to the console, confirming that data is flowing correctly before we add complexity.

The Collector's behavior is defined by a YAML configuration file. For this initial setup, you need to understand three top-level sections: receivers, exporters, and service.

Receivers

Receivers are the entry points for all telemetry data coming into the Collector from your applications and infrastructure.

They're configured to ingest data in various ways such as listening for network traffic, actively polling endpoints, reading from local sources (like files), or querying infrastructure APIs.

For example, the OTLP receiver sets up an endpoint that accepts data sent using the OpenTelemetry Protocol, while the Prometheus receiver periodically scrapes metrics from specified targets.

Exporters

Exporters are the final destinations for all telemetry data leaving the Collector after it has been processed.

They're responsible for translating data into the required format and transmitting it to various backend systems, such as observability platforms, databases, or message queues.

For example, the otlphttp exporter can send data to any OTLP-compatible backend over HTTP, while the debug exporter simply writes telemetry data to the console for debugging.

Service

The service section is the central orchestrator that activates and defines the flow of data through the Collector. No component is active unless it is enabled here.

It works by defining pipelines for each signal type (traces, metrics, or logs). Each pipeline specifies the exact path data will take by linking receivers, processors, and exporters.

For example, a traces pipeline could be configured to receive span data over OTLP, and fan it out to Jaeger through the OTLP exporter.

To see the three components in action, let's create our first configuration file:

# otelcol.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [debug]

This configuration creates a simple pipeline for logs alone. It sets up an otlp receiver to accept log data sent over GRPC on port 4317. Any logs received are immediately passed without any processing to the debug exporter, which then prints the full, detailed content to the Collector's stderr.

To test your pipeline, you need an application that can generate and send telemetry data. A convenient tool for this is otelgen, which produces synthetic logs, traces, and metrics.

You can define and run the Collector and the otelgen tool using the following Docker Compose configuration:

# docker-compose.yml
services:
  otelcol:
    image: otel/opentelemetry-collector-contrib:0.129.1
    container_name: otelcol
    volumes:
      - ./otelcol.yaml:/etc/otelcol-contrib/config.yaml
    restart: unless-stopped

  otelgen:
    image: ghcr.io/krzko/otelgen:v0.5.2
    container_name: otelgen
    command:
      [
        "--otel-exporter-otlp-endpoint",
        "otelcol:4317",
        "--insecure",
        "logs",
        "multi",
      ]
    depends_on:
      - otelcol

networks:
  otelnet:
    driver: bridge

The otelgen service is configured via its command arguments to send telemetry that matches our Collector's setup:

--otel-exporter-otlp-endpoint otelcol:4317: Tells otelgen to send data to the otelcol service on port 4317.
--insecure: Disables TLS.
logs: Instructs otelgen to generate log data specifically.
multi: A subcommand that generates a continuous, varied stream of logs.

To see it in action, start both services in detached mode:

docker compose up -d

Once running, the otelcol service listens for OTLP data over gRPC on port 4317, and the otelgen service generates and sends a continuous stream of logs to it.

You can monitor the Collector's output to verify that it's receiving the logs:

docker compose logs otelcol -f --no-log-prefix

You will see a continuous stream of log data being printed to the console. A single log entry will be formatted like this, showing rich contextual information like the severity, body, and various attributes:

ResourceLog #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.26.0
Resource attributes:
     -> host.name: Str(node-1)
     -> k8s.container.name: Str(otelgen)
     -> k8s.namespace.name: Str(default)
     -> k8s.pod.name: Str(otelgen-pod-ab06ca8b)
     -> service.name: Str(otelgen)
ScopeLogs #0
ScopeLogs SchemaURL:
InstrumentationScope otelgen
LogRecord #0
ObservedTimestamp: 2025-07-06 11:21:57.085421018 +0000 UTC
Timestamp: 2025-07-06 11:21:57.085420886 +0000 UTC
SeverityText: Error
SeverityNumber: Error(17)
Body: Str(Log 3: Error phase: finish)
Attributes:
     -> worker_id: Str(3)
     -> service.name: Str(otelgen)
     -> trace_id: Str(46287c1c7b7eebea22af2b48b97f4a49)
     -> span_id: Str(f5777521efe11f94)
     -> trace_flags: Str(01)
     -> phase: Str(finish)
     -> http.method: Str(PUT)
     -> http.status_code: Int(403)
     -> http.target: Str(/api/v1/resource/3)
     -> k8s.pod.name: Str(otelgen-pod-8f215fc5)
     -> k8s.namespace.name: Str(default)
     -> k8s.container.name: Str(otelgen)
Trace ID:
Span ID:
Flags: 0

Understanding the `debug` exporter output

The output from the debug exporter shows the structured format of OpenTelemetry data (OTLP). It's hierarchical, starting from the resource that generated the telemetry all the way down to the individual telemetry record. Let's break down what you're seeing.

ResourceLogs and Resource attributes

ResourceLog #0: This is the top-level container. The #0 indicates it's the first resource in this batch, which means all telemetry within this block comes from the same resource.
Resource attributes: These are key-value pairs that describe the entity that produced the log. This could be a service, a container, or a host machine. In the example, attributes like service.name and k8s.pod.name apply to every log generated by this resource.

ScopeLogs

ScopeLogs #0: Within a resource, telemetry is grouped by its origin, known as the instrumentation scope. This block contains a batch of logs from the same scope.
InstrumentationScope: This identifies the specific library or module that generated the log (in this case, otelgen). This is useful for knowing which part of your application emitted the log.

LogRecord

Within a single ResourceLog block, you may see multiple LogRecord entries (#0, #1, #2, and so on), all belonging to the same resource.

LogRecord #0: This is the first log entry belonging to the resource. The key fields are:
- Timestamp: When the event occurred.
- SeverityNumber / SeverityText: The log level, such as ERROR or INFO.
- Body: The actual log message content.
- Attributes: Key-value pairs that provide context specific to this single log event.
- Trace ID / Span ID: When populated, they directly link a log to a specific trace and span, allowing you to easily correlate logs and traces in your observability backend.

Congratulations, you've built and verified your first telemetry pipeline! It's simple, but it establishes the fundamental flow of data from a source, through the Collector, and to an exit point. Now, let's make it more powerful.

Processing and transforming telemetry

Right now, your pipeline is just an empty conduit. Data goes in one end and comes out the other untouched. The real power of the Collector lies in its ability to process data in-flight. This is where processors come in.

Processors are intermediary components in a pipeline that can inspect, modify, filter, or enrich your telemetry. Let's add a few essential processors to solve common problems and make the pipeline more intelligent.

Our new pipeline flow will look like this: Receiver -> [Processors] -> Exporter.

Batching telemetry for efficiency

Sending every single span or metric individually over the network is incredibly inefficient. It creates high network traffic and puts unnecessary load on the backend. The batch processor solves this by grouping telemetry into batches before exporting.

Go ahead and add it to your processors section. By default, it buffers data for a short period to create batches automatically:

# otelcol.yaml
# Add this top-level 'processors' section
processors:
  batch:
    # You can customize the default values for more control
    # send_batch_size: 8192
    # timeout: 200ms

service:
  pipelines:
    logs:
      receivers: [otlp]
      # Add the processor to your pipeline's execution path.
      # Order matters here if you have multiple processors.
      processors: [batch]
      exporters: [debug]

With this simple addition, your pipeline now buffers data for up to 200 milliseconds or until it has 8192 items (whichever comes first) before it forwards data to the configured exporters.

Reducing noise by filtering telemetry

Telemetry data can be noisy. For example, frequent DEBUG level logs are often useful in development but are often superfluous in production except when debugging an active issue. Let's add a bouncer to our pipeline to drop this noise at the source.

We'll use the filter processor, which lets you drop telemetry data using the powerful OpenTelemetry Transformation Language (OTTL). Let's say you wanted to drop all logs below the INFO severity level, you can do so with the following modifications:

# otelcol.yaml
processors:
  batch:
  # The filter processor lets you exclude telemetry data based on its attributes
  filter:
    logs:
      log_record:
        - severity_number < SEVERITY_NUMBER_INFO

service:
  pipelines:
    logs:
      receivers: [otlp]
      # The order is important. You want to drop data before batching it.
      processors: [filter, batch]
      exporters: [debug]

Now, any log with a severity number less than 9 (INFO) will be dropped by the Collector and will never reach the debug exporter.

Modifying and enriching telemetry data

When you'd like to add, remove, or modify attributes in your telemetry data, there are a few general-purpose processors you can use:

resource processor: For actions targeting resource-level attributes (e.g., host.name, service.name).
attributes processor: For manipulating attributes of individual logs, spans, or metric datapoints.
transform processor: The most powerful of the three, for performing complex transformations on any part of your telemetry data.

Some common use cases for these processors include:

Redacting or removing sensitive information from telemetry before it leaves your systems.
Enriching data by adding static attributes.
Renaming or standardizing attributes to conform to semantic conventions across different services.
Correcting malformed or misplaced data sent by older or misconfigured instrumentation.

Let's examine the structure of the OTLP log records being sent by the otelgen tool once again:

ResourceLog #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.26.0
Resource attributes:
     -> host.name: Str(node-1)
     -> k8s.container.name: Str(otelgen)
     -> k8s.namespace.name: Str(default)
     -> k8s.pod.name: Str(otelgen-pod-b9919c90)
     -> service.name: Str(otelgen)
LogRecord #0
ObservedTimestamp: 2025-07-03 16:01:40.264711241 +0000 UTC
Timestamp: 2025-07-03 16:01:40.264711041 +0000 UTC
SeverityText: Fatal
SeverityNumber: Fatal(21)
Body: Str(Log 1763: Fatal phase: finish)
Attributes:
     -> worker_id: Str(1763)
     -> service.name: Str(otelgen)
     -> trace_id: Str(a85d432127e63d667508563efd73af52)
     -> span_id: Str(34c07d59e6cfa2d9)
     -> trace_flags: Str(01)
     -> phase: Str(finish)
     -> http.method: Str(POST)
     -> http.status_code: Int(200)
     -> http.target: Str(/api/v1/resource/1763)
     -> k8s.pod.name: Str(otelgen-pod-b9919c90)
     -> k8s.namespace.name: Str(default)
     -> k8s.container.name: Str(otelgen)
Trace ID:
Span ID:
Flags: 0

There are (at least) three issues here that deviate from the correct OpenTelemetry data model and semantic conventions:

Misplaced trace context: The trace_id, span_id, and trace_flags values are incorrectly placed inside the Attributes map, while the dedicated top-level Trace ID, Span ID, and Flags fields are empty.
Redundant attributes: Resource attributes like k8s.pod.name and service.name are duplicated in the log record's Attributes.
Deprecated attributes: HTTP attributes like http.method, http.target, and http.status_code have all been deprecated in favor of newer attributes.

The transform processor is the perfect tool for fixing these issues. Add the following modifications to your otelcol.yaml file:

# otelcol.yaml
processors:
  transform:
    log_statements:
      # Move trace context from attributes to the correct top-level fields
      - context: log
        statements:
          - set(trace_id.string, attributes["trace_id"])
          - set(span_id.string, attributes["span_id"])
          - set(flags, Int(attributes["trace_flags"]))
      # Delete the original, now redundant, trace context attributes
      - context: log
        statements:
          - delete_key(attributes, "trace_id")
          - delete_key(attributes, "span_id")
          - delete_key(attributes, "trace_flags")
      # Delete the duplicated resource attributes from the log record's attributes
      - context: log
        statements:
          - delete_key(attributes, "k8s.pod.name")
          - delete_key(attributes, "k8s.namespace.name")
          - delete_key(attributes, "k8s.container.name")
          - delete_key(attributes, "service.name")

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [filter, transform, batch] # Add the transform processor to the pipeline
      exporters: [debug]

This configuration uses OTTL statements to clean up the log records:

set(trace_id, ...): This function takes the value from the trace_id key within the attributes map and sets it as the top-level Trace ID for the log record. The same logic applies to the other statements.
delete_key(attributes, ...): After moving the values, this function removes the original keys from the attributes map to eliminate redundancy.

You can recreate the containers to see it in action:

docker compose up --force-recreate -d

When you check the logs, you'll notice that the outgoing log data is now correctly formatted, smaller in size, and fully compliant with semantic conventions with the Trace ID and Span ID fields properly populated:

2025-07-03T16:36:49.418Z     info    ResourceLog #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.26.0
Resource attributes:
     -> host.name: Str(node-1)
     -> k8s.container.name: Str(otelgen)
     -> k8s.namespace.name: Str(default)
     -> k8s.pod.name: Str(otelgen-pod-3efafa6f)
     -> service.name: Str(otelgen)
ScopeLogs #0
ScopeLogs SchemaURL:
InstrumentationScope otelgen
LogRecord #0
ObservedTimestamp: 2025-07-03 16:36:48.41161663 +0000 UTC
Timestamp: 2025-07-03 16:36:48.411616563 +0000 UTC
SeverityText: Error
SeverityNumber: Error(17)
Body: Str(Log 38: Error phase: finish)
Attributes:
     -> worker_id: Str(38)
     -> phase: Str(finish)
     -> url.path: Str(/api/v1/resource/340)
     -> http.response.status_code: Int(200)
     -> http.request.method: Str(GET)
Trace ID: 86713e2736d6f6a398047b9317b11398
Span ID: d06e86785766aa64
Flags: 1

Ensuring resilience with the Memory Limiter

An overloaded service could suddenly send a massive flood of data, overwhelming the Collector and causing it to run out of memory and crash. This would create a total visibility outage.

The memory_limiter processor acts as a safety valve to prevent this. It monitors memory usage and starts rejecting data if it exceeds a configured limit, enforcing backpressure on the data source.

# otelcol.yaml
processors:
  batch:
  filter: # ...
  transform: # ...
  memory_limiter:
    # How often to check the collector's memory usage.
    check_interval: 1s
    # The hard memory limit in Mebibytes (MiB). If usage exceeds this,
    # the collector will start rejecting new data.
    limit_mib: 400
    # A soft limit. When usage drops below this, the collector will
    # start accepting data again.
    spike_limit_mib: 100

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [memory_limiter, filter, transform, batch]
      exporters: [debug]

Note that the memory_limiter should come first in your pipeline's processor list. If it's over the limit, you want to reject data immediately, before wasting CPU cycles on other processing.

Handling multiple signals with parallel pipelines

So far, you've built a simple pipeline for processing logs. However, a key strength of the OpenTelemetry Collector is its ability to manage all signals simultaneously within a single instance. You can achieve this by defining parallel pipelines, one for each signal type, in the service section.

Let's expand the configuration to also process traces. The goal is to receive traces from an application, batch them for efficiency, and then send them to a Jaeger instance for visualization, while the existing logs pipeline continues to operate independently (writing to the console as before).

To send traces to Jaeger, you can use the OTLP exporter. You can provide an identifying name for any component using the type/name syntax as follows:

# otelcol.yaml
exporters:
  otlp/jaeger:
    endpoint: jaeger:4317 # The address of the Jaeger gRPC endpoint
    tls:
      insecure: true # Use TLS in production

You can now add a new pipeline to the service section specifically for traces. This pipeline will:

Reuse the same otlp receiver we already defined.
Reuse the batch and memory_limiter processors.
Send its data to the new otlp/jaeger exporter.

Here is the complete service section showing both the logs and traces pipelines running in parallel:

# otelcol.yaml
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [memory_limiter, filter, transform, batch]
      exporters: [debug]

    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/jaeger]

To test this, you need to update your docker-compose.yml to run a Jaeger instance and a second otelgen service configured to generate traces:

# docker-compose.yml
services:
  otelcol:
    image: otel/opentelemetry-collector-contrib:0.129.1
    container_name: otelcol
    volumes:
      - ./otelcol.yaml:/etc/otelcol-contrib/config.yaml
    restart: unless-stopped

  jaeger:
    image: jaegertracing/all-in-one:1.71.0
    container_name: jaeger
    ports:
      - 16686:16686

  otelgen-logs:
    image: ghcr.io/krzko/otelgen:v0.5.2
    container_name: otelgen-logs
    command:
      [
        "--otel-exporter-otlp-endpoint",
        "otelcol:4317",
        "--insecure",
        "logs",
        "multi",
      ]
    depends_on:
      - otelcol

  otelgen-traces:
    image: ghcr.io/krzko/otelgen:v0.5.2
    container_name: otelgen-traces
    command:
      [
        "--otel-exporter-otlp-endpoint",
        "otelcol:4317",
        "--insecure",
        "--duration",
        "86400",
        "traces",
        "multi",
      ]
    depends_on:
      - otelcol

networks:
  otelnet:
    driver: bridge

Notice we now have two otelgen services: otelgen-logs sends logs as before, and otelgen-traces sends traces to the same OTLP endpoint on our Collector.

Recreate the containers with the updated configuration:

docker compose up --force-recreate --remove-orphans -d

While the otelcol logs will continue showing the processed logs, the easiest way to verify the traces pipeline is to check the Jaeger UI.

Open your web browser and navigate to http://localhost:16686. In the Jaeger UI, select otelgen from the Service dropdown menu and click Find Traces.

You will see a list of traces generated by the otelgen-traces service, confirming that your new traces pipeline is successfully receiving, processing, and exporting trace data to Jaeger.

With this setup, you have a single Collector instance efficiently managing two completely separate data flows, demonstrating the power and flexibility of defining multiple pipelines.

Fanning out to multiple destinations

A key advantage of the OpenTelemetry Collector is its ability to easily route telemetry to multiple destinations at once, a concept often called "fanning out". This is done by simply adding more exporters to a pipeline's exporters list.

Let's demonstrate this by forwarding both our logs and traces to Dash0, an OpenTelemetry-native platform, in addition to the existing destinations.

You'll need to sign up for a free trial first, find the OpenTelemetry Collector integration, and copy your authentication token and Dash0 endpoint (OTLP via gRPC) into your configuration file:

# otelcol.yaml
exporters:
  # [...]
  otlp/dash0:
    endpoint: <your_dash0_endpoint>
    headers:
      Authorization: Bearer <your_dash0_token>

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [memory_limiter, filter, transform, batch]
      exporters: [debug, otlp/dash0]

    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/jaeger, otlp/dash0]

With this change, both pipelines now fan out the processed data to the specified exporters. You'll see the data in your Dash0 dashboard as follows:

This capability allows you to experiment with multiple backends, migrate between vendors without downtime, or satisfy other uses for your telemetry without touching your application code.

Chaining pipelines with connectors

You can create powerful data processing flows by generating new telemetry signals from your existing data.

This is possible with connectors. A connector is a special component that acts as both an exporter for one pipeline and a receiver for another, allowing you to chain pipelines together.

Let's demonstrate this by building a system that generates an error count metric from the otelgen log data. The count connector is perfect for this.

First, you'll need to define the count connector and configure it to create a metric named log_error.count that increments every time it sees a log with a severity of ERROR or higher:

# otelcol.yaml
connectors:
  count/log_errors:
    logs:
      log_error.count:
        description: count of errors logged
        conditions:
          - severity_number >= SEVERITY_NUMBER_ERROR

To use this, go ahead and update your service configuration to create a new metrics pipeline. The count/log_errors connector will serve as the bridge: it will be an exporter for the logs pipeline and a receiver for the new metrics pipeline:

# otelcol.yaml
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [memory_limiter, filter, transform, batch]
      # The connector is added as a destination for logs.
      exporters: [debug, otlp/dash0, count/log_errors]

    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/jaeger, otlp/dash0]

    metrics:
      # This new pipeline receives data exclusively from the connector.
      receivers: [count/log_errors]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/dash0]

This configuration is a game-changer because it allows you to derive new insights from existing data streams directly within the Collector. The data flow is now:

The logs pipeline processes logs and sends a copy to the count/log_errors connector.
The count/log_errors connector inspects these logs, generates a new log_error.count metric based on our condition, and passes this metric along. sends them to your backend.
The metrics pipeline receives the newly generated metric, batches it, and sends them to your backend.

After relaunching the services, you'll see the new log_error.count metric appear in your dashboard, all without adding a single line of metrics instrumentation code to your application.

This is a basic example, but it demonstrates the power of a true pipeline architecture. The same principle can be used for more advanced scenarios, like using the spanmetrics connector to automatically generate full RED metrics (request rates, error counts, and duration histograms) directly from your trace data.

Understanding Collector distributions

When you use the OpenTelemetry Collector, you're not running a single, monolithic application. Instead, you use a distribution: a specific binary packaged with a curated set of components (receivers, processors, exporters, and extensions).

This model exists to allow you to use a version of the Collector that is tailored to your specific needs or even create your own. There are three primary types of distributions you will encounter:

1. Official OpenTelemetry distributions

The OpenTelemetry project maintains several official distributions. The two most common are:

Core (otelcol): This is a minimal, lightweight distribution that includes only the most essential and stable components. It provides a stable foundation but has limited functionality.
Contrib (otelcol-contrib): This is the most comprehensive version, that includes almost every component from both the core and contrib repositories. It is the recommended distribution for getting started, as it provides the widest range of capabilities for connecting to various sources and destinations without needing to build a custom version.

2. Vendor distributions

Some observability vendors provide their own Collector distributions. These are typically based on the otelcol-contrib distribution but are pre-configured with the vendor's specific exporter and other recommended settings. Using a vendor distribution can simplify the process of sending data to that vendor's platform.

3. Custom distributions

For production environments, the recommended practice is to build your own custom distribution. This involves creating a lightweight, fit-for-purpose Collector binary that contains only the components you need.

You can create a custom distribution using the OpenTelemetry Collector Builder (ocb) tool. It involves creating a simple manifest file that lists the components you want to include, and then running the ocb tool to compile your custom binary.

You can learn more about building a custom Collector distribution by reading this guide.

Debugging and observing your pipeline

A critical piece of infrastructure like your telemetry pipeline must itself be observable and easy to debug. If the Collector is dropping data, experiencing high latency, or is unhealthy, you definitely need to know about it.

Fortunately, the Collector is instrumented out-of-the-box and provides several tools for validation and observation.

Validating your configuration

Before deploying the Collector, you should always validate that your config.yaml file is syntactically correct. The primary way to do this is with the validate subcommand which checks the configuration file for errors without starting the full Collector service:

otelcol-contrib validate --config=otelcol.yaml

If the configuration is valid, the command will exit silently. If there are errors, it will print them to the console:

You can also use OtelBin to visualize your pipeline, and validate it against various Collector distributions before you deploy.

If you're complex OTTL statements in the transform or filter processor, you will also find the OTTL Playground to be a useful resource for understanding how different configurations impact the OTLP data transformation.

Live debugging

When building a pipeline, you'll often need to inspect the data flowing through it in real-time. As you've already seen, the debug exporter is primary way to do this.

By adding it to any pipeline's exporters list, you can print the full content of traces, metrics, or logs to the console, and verify that your receivers and processors are working as expected.

For debugging the Collector components themselves, you can enable the zPages extension:

# otelcol.yaml
extensions:
  zpages: # default endpoint is localhost:55679

service:
  extensions: [zpages]
  pipelines:
    # ... your pipelines

Once the Collector is running, you can access several useful debugging pages in your browser, such as /debug/pipelinez to view your pipeline components or /debug/tracez to see recently sampled traces.

Observing the Collector's internal telemetry

In production environments, you'll need to monitor the Collector's health and performance over time. This is configured under the service.telemetry section.

By default, the Collector sends its own internal logs to stderr, and its often the first place you'll check when there's a problem with your pipeline. For metrics, the Collector can expose its own data in a Prometheus-compatible format:

# otelcol.yaml
service:
  telemetry:
    metrics:
      readers:
        - pull:
            exporter:
              prometheus:
                host: "0.0.0.0"
                port: 8888

You can now scrape this endpoint with a Prometheus instance to monitor key health indicators like otelcol_exporter_send_failed_spans_total, otelcol_processor_batch_send_size, and otelcol_receiver_accepted_spans.

You can also push the metrics to an OTLP-compatible backend using the following configuration:

# otelcol.yaml
service:
  telemetry:
    metrics:
      readers:
        - periodic:
            exporter:
              otlp:
                protocol: http/protobuf
                endpoint: https://backend:4318

For more details, see the official documentation on Collector telemetry.

Going to production: Collector deployment patterns

How you run the Collector in production is a critical architectural decision. Your deployment strategy affects the scalability, security, and resilience of your entire observability setup. The two fundamental roles a Collector can play are that of an agent or a gateway, which can be combined into several common patterns.

1. Agent-only deployment

The simplest pattern is to deploy a Collector agent on every host or as a sidecar to every application pod. In this model, each agent is responsible for collecting, processing, and exporting telemetry directly to one or more backends.

Application → OpenTelemetry Collector (Agent) → Observability Backend

This approach is easy to start with but it offers limited durability, as agents typically buffer in memory, meaning a single node failure can lead to data loss.

2. Agent and gateway deployment

A more robust production pattern enhances the agent deployment with a new, centralized gateway layer. In this model, the agent's role is simplified: it handles local collection and metadata enrichment before forwarding all telemetry to the gateway.

This gateway is a standalone, centralized service consisting of one or more Collector instances that receive telemetry from all agents. It's the ideal place for heavy processing like PII scrubbing, filtering, and tail-based sampling, which ensures rules are applied consistently before data leaves your environment.

Application → Collector (Agent) → Collector (Gateway) → Observability Backend

This layered approach provides the best of both worlds: agents handle local collection and metadata enrichment efficiently, while the gateway provides centralized control, security, and processing.

High-scale deployment with a message queue

When you're dealing with massive data volumes or require extreme durability, the standard pattern is to introduce an event queue (like Apache Kafka) between your agents and a fleet of Collectors that act as consumers.

Application → Collector (Agent) → Message Queue → Collector (Aggregator) → Backend(s)

This pattern provides two advantages. The message queue acts as a massive buffer for durability; even if the aggregator fleet is down, agents can continue sending data to the queue, preventing data loss.

It also provides load-leveling by decoupling the agents from the aggregators, which smooths out traffic spikes and allows the aggregators to consume data at a steady rate.

You're now a pipeline architect

We've journeyed from a simple data pass-through to a powerful, multi-stage pipeline that enriches, filters, and routes telemetry data, even generating new, valuable signals along the way.

By adopting the pipeline mindset, you gain:

Centralized control, allowing you to manage your entire telemetry flow from one place.
Vendor neutrality so you can swap backends with a simple config change or use multiple vendors at once.
Efficiency and cost savings by applying consistent filtering policies across your entire environment which ultimately reduces your observability bill.
Enhanced security by scrubbing sensitive data before it ever leaves your infrastructure.
Powerful capabilities using advanced patterns like metric generation that would be complex or impossible otherwise.

To complete the picture, you need to send that data to an OpenTelemetry-native observability platform like Dash0 that makes it easy to quickly move from raw data to insight.

For more on the Collector itself, refer to the official documentation. Thanks for reading!

Production-Grade Python Logging Made Easier with Loguru

Ayooluwa Isaiah — Wed, 05 Nov 2025 09:19:47 +0000

Logs are often the first place you look when something goes wrong in production. They're your running commentary of what the application is doing and why, and when it fails to do what it's supposed to.

While the standard logging module can be configured to produce high-quality telemetry, achieving this requires significant boilerplate: custom formatters, filters, handlers, and complex YAML configurations. It's powerful, but it's not simple.

So what if you could achieve the same structured, contextual, and production-ready logging with a fraction of the complexity?

This is the promise of Loguru. It's a logging library designed from the ground up to replace the cumbersome setup of the standard library with a simple, unified API that supports modern observability practices.

This guide will directly address the patterns and pain points of the standard logging module and show how Loguru simplifies them without compromising on effectiveness and flexibility.

By the end, you will have a lean, modern logging setup that feels natural to use and is ready for production.

Let's get started!

Understanding the Loguru philosophy

With Python's built-in logging module, the common pattern is to configure everything once by setting up your handlers, formatters, and filters, and then, in each module, grab a namespaced logger with logging.getLogger(__name__).

It's a solid system, but the initial configuration can feel heavy: multiple objects to wire together, YAML or dictConfig to maintain, and a lot of moving parts if you just want to get clean logs quickly.

Loguru takes a far simpler approach. Instead of asking you to set up a hierarchy of loggers, it gives you one ready-to-go logger that you just import (after you install it first):

from loguru import logger

logger.info("Hello, Loguru!")

That's it. The logger you imported is configured to print colorized messages to the stderr, complete with timestamps, log levels, module names, and line numbers:

2025-09-02 13:53:03.686 | INFO     | __main__:<module>:3 - Hello, Loguru!

While this is great for local development and is easy to read at a glance, especially with the colors, you'll want something more machine-friendly in production environments such as JSON.

Start by removing the default stderr handler so you can make a fresh start:

from loguru import logger

logger.remove() # remove the default configuration

Then add your own sink with the exact behavior you want. In Loguru, a sink is simply a destination for your logs. It could be the standard output, a file path, a custom function, or even a logging.Handler from the standard library.

Instead of wiring up separate handlers, formatters, and filters, you'll configure everything in one place with a single call to logger.add():

logger.add(sys.stdout, level="INFO", serialize=True)

That single add() call completely defines the sink: where logs go, how they're formatted, and which levels get through. The serialize argument is what causes the output to be formatted as a JSON object, which looks something like this:

{
  "text": "2025-09-02 17:08:04.498 | INFO     | __main__:<module>:8 - Application started\n",
  "record": {
    "elapsed": { "repr": "0:00:00.004885", "seconds": 0.004885 },
    "exception": null,
    "extra": {},
    "file": {
      "name": "main.py",
      "path": "/Users/ayo/dev/dash0/demo/loguru-demo/main.py"
    },
    "function": "<module>",
    "level": { "icon": "ℹ", "name": "INFO", "no": 20 },
    "line": 8,
    "message": "Application started",
    "module": "main",
    "name": "__main__",
    "process": { "id": 65239, "name": "MainProcess" },
    "thread": { "id": 8437194496, "name": "MainThread" },
    "time": {
      "repr": "2025-09-02 17:08:04.498902+02:00",
      "timestamp": 1756825684.498902
    }
  }
}

This structure is deliberately rich, and it includes everything Loguru knows about the log event, including timestamps, process and thread IDs, the module and line number, even elapsed time since the program started.

If that feels too heavy for your needs, you don't have to stick with the default. Loguru lets you provide a custom serializer function to control exactly how log records are turned into JSON. That way, you can keep the fields that matter and drop the rest.

To do this reliably, you need a two-step process to avoid conflicts with Loguru's internal formatter. First, you'll define a function that serializes your log record into the desired JSON format. Second, you'll use Loguru's patch() method to add this JSON string as a new field to the record. Finally, you'll tell the sink to only output that new field through format:

import sys
import json
from loguru import logger
import traceback

logger.remove()

def serialize(record):
    subset = {
        "time": record["time"].isoformat(),
        "level": record["level"].name,
        "message": record["message"],
    }
    # Merge extra fields directly into the top-level dict
    if record["extra"]:
        subset.update(record["extra"])

    if record["exception"]:
        exc = record["exception"]
        subset["exception"] = {
            "type": exc.type.__name__,
            "value": str(exc.value),
            "traceback": traceback.format_exception(exc.type, exc.value, exc.traceback),
        }

    return json.dumps(subset)

def patching(record):
    record["serialized"] = serialize(record)

logger = logger.patch(patching)

logger.add(sys.stdout, level="INFO", format="{serialized}")

logger.info("Application started")

This produces a much slimmer JSON log:

{
  "time": "2025-09-02T17:58:09.538713+02:00",
  "level": "INFO",
  "message": "Application started"
}

Note that all following examples in this guide will assume that you're using this custom serializer. For further customization, you can find all the available record fields in the Loguru documentation.

How log levels work in Loguru

Just like the standard logging module, Loguru uses log levels to annotate the severity and control the verbosity of your logs. Levels let you decide which messages are worth keeping and which ones to ignore, especially once your application is running in production.

Out of the box, Loguru supports the familiar set of levels:

Level	Numeric value	Typical use case
`TRACE`	5	Ultra-verbose debugging (finer than `DEBUG`).
`DEBUG`	10	Detailed diagnostics for developers.
`INFO`	20	Normal application events.
`SUCCESS`	25	A Loguru-specific level, for happy paths.
`WARNING`	30	Potential problems worth attention.
`ERROR`	40	An operation failed.
`CRITICAL`	50	Severe failures, system at risk.

Notice that Loguru adds two extra levels compared to the standard library:

TRACE for when even DEBUG isn't enough.
SUCCESS as a positive counterpart to WARNING or which could be handy for marking milestones.

You can log at these levels using corresponding methods on the logger:

logger.trace("Function entered")
logger.debug("Fetching user details")
logger.info("Application started")
logger.success("Background job completed")
logger.warning("Cache miss")
logger.error("Database update failed")
logger.critical("Out of memory!")

If you ever need to create a custom level, you can do so with the level() method:

logger.level("FATAL", no=60, color="<red><bold>", icon="!!!")

Then you can use the generic log() method and provide the custom level's name:

logger.log("FATAL", "Out of memory!")

Controlling which logs appear

Every sink that's added with logger.add() can be given a minimum level. Messages below that threshold are dropped before they're written. For example:

import sys
from loguru import logger

logger.remove()
logger.add(sys.stdout, level="WARNING")

logger.debug("Debug message")    # ignored
logger.info("Informational")     # ignored
logger.warning("A warning")      # logged
logger.error("An error")         # logged

A common pattern is using environment variables to set the level so you can bump verbosity up or down without touching the code.

Loguru doesn't read environment variables automatically, but you can grab the value yourself (with os.getenv) and feed it into logger.add():

import os

log_level = os.getenv("LOG_LEVEL", "INFO").upper()

logger.add(sys.stdout, level=log_level)

Now you can control logging through the LOG_LEVEL variable:

LOG_LEVEL=WARNING python main.py

Effortless contextual logging

A common pain point with logging is the lack of context. A message like "Failed to update record" doesn't tell you much on its own. Which record? For which user? During which request? Without those details, you're left guessing.

With Loguru, you don't have to cram all of that context into the message string itself. Every logging call can include extra key--value pairs, and they'll automatically be attached to the record.

logger.error("Failed to update record", user_id="usr-1234", record_id="rec-9876")

Assuming you're using the custom JSON serializer shown earlier, you will observe the following output:

{
  "time": "2025-09-02T19:23:17.150211+02:00",
  "level": "ERROR",
  "message": "Failed to update record",
  "user_id": "usr-1234",
  "record_id": "rec-9876"
}

This approach works fine for adding one-off contextual fields to a log record. But if you need certain dynamic per-request fields (like a correlation ID) to appear across multiple fields, it would be tedious to pass this information around manually.

That's where Loguru's bind() method comes in. With it, you can attach context to a logger once, and it will automatically carry through to every log call that uses it.

Here's an example:

request_logger = logger.bind(request_id="req-42", user_id="usr-1234")

request_logger.info("Fetching user profile")
request_logger.error("Failed to update record")

Both of the resulting log entries will include the request_id and user_id without you having to repeat them each time:

{"time": "2025-09-02T20:26:37.400265+02:00", "level": "INFO", "message": "Fetching user profile", "request_id": "req-42", "user_id": "usr-1234"}
{"time": "2025-09-02T20:26:37.400330+02:00", "level": "ERROR", "message": "Failed to update record", "request_id": "req-42", "user_id": "usr-1234"}

While bind() works well for adding context to logs in the same scope, it's often more useful to attach context for the duration of a block of code, such as the lifetime of an HTTP request in a web app.

That's where logger.contextualize() comes in. It's a context manager that pushes values into the logging context when you enter the block, and automatically removes them when you exit.

import sys
import json
from loguru import logger
import os
from fastapi import FastAPI, Request
import uuid
import time

# ...existing logging configuration

app = FastAPI()

@app.middleware("http")
async def add_request_context(request: Request, call_next):
    start_time = time.monotonic()

    request_id = request.headers.get("X-Request-ID", str(uuid.uuid4()))
    client_ip = request.headers.get("X-Forwarded-For", request.client.host)
    user_agent = request.headers.get("user-agent", "-")

    with logger.contextualize(request_id=request_id):
        logger.info(
            "Incoming {method} request to {path}",
            method=request.method,
            path=request.url.path,
            client_ip=client_ip,
            user_agent=user_agent,
        )

        response = await call_next(request)

        duration_ms = (time.monotonic() - start_time) * 1000

        log_level = "INFO"

        if response.status_code >= 500:
            log_level = "ERROR"
        elif response.status_code >= 400:
            log_level = "WARNING"

        logger.log(
            log_level,
            "{method} {path} completed with status {status} in {duration:.2f} ms",
            method=request.method,
            path=request.url.path,
            status=response.status_code,
            duration=duration_ms,
        )

        return response

@app.get("/users/{user_id}")
async def get_user(user_id: str):
    with logger.contextualize(user_id=user_id):
        logger.info("User profile request received.")

    return {"user_id": user_id}


if __name__ == "__main__":
    import uvicorn

    uvicorn.run(
        "main:app",
        host="0.0.0.0",
        port=8000,
        reload=False,
        access_log=False,
    )

With contextualize(), you only need to declare the context once at the start of the request. The middleware guarantees that every log line for that request will have the right identifying fields, and they'll disappear as soon as the request finishes.

The result is clean, consistent, and scoped contextual logging which is exactly what you need to correlate events in production without cluttering your log calls.

Once you send some requests to that route, you'll notice how each log includes the same request_id:

{..., "message": "Incoming GET request to /users/12", "request_id": "598e2e5f-e33d-4f05-a658-0a8287d766a6", "method": "GET", "path": "/users/12", "client_ip": "127.0.0.1", "user_agent": "curl/8.7.1"}
{..., "message": "User profile request received.", "request_id": "598e2e5f-e33d-4f05-a658-0a8287d766a6", "user_id": "12"}
{..., "message": "GET /users/12 completed with status 200 in 1.80 ms", "request_id": "598e2e5f-e33d-4f05-a658-0a8287d766a6", "method": "GET", "path": "/users/12", "status": 200, "duration": 1.7961669946089387}

Error and exception logging with Loguru

The simplest way to capture errors in Loguru is by calling logger.error() and including the exception details if you're in an except block:

from loguru import logger

try:
    1 / 0
except Exception as e:
    logger.error("Something went wrong: {}", e)

This will log the error message (division by zero), but not the full traceback which is a crucial piece of context that'll help you debug the issue.

To capture the full exception, you'll need to use the logger.exception() method. It exposes a richer record["exception"] object which includes the error type, value, and complete Python traceback object:

try:
    1 / 0
except Exception as e:
    logger.exception("Something went wrong: {}", e)

This logs the message at ERROR level and includes the full traceback. If you're using our custom JSON serializer, you'll see the fields in a structured form:

{
  "time": "2025-09-03T07:55:37.157368+02:00",
  "level": "ERROR",
  "message": "Something went wrong: division by zero",
  "exception": {
    "type": "ZeroDivisionError",
    "value": "division by zero",
    "traceback": [
      "Traceback (most recent call last):\n",
      "  File \"/Users/ayo/dev/dash0/demo/loguru-demo/main.py\", line 50, in <module>\n    1 / 0\n    ~~^~~\n",
      "ZeroDivisionError: division by zero\n"
    ]
  }
}

You might also see a colorized traceback in your terminal as follows despite serializing to JSON:

This happens because Loguru tries to be helpful: if a log record contains an exception but the sink's format string doesn't explicitly handle it (with {exception}), Loguru appends the formatted traceback by default.

To fix this, you can use a custom function to define the format of the logs as follows:

def custom_formatter(record):
    return "{serialized}\n"

logger.add(sys.stdout, level="INFO", format=custom_formatter)

Even though we're simply returning the same {serialized} format, using the custom_formatter() function tells Loguru to output exactly and only the content of our pre-formatted JSON string, so that the colorized error log will no longer appear in the console.

Using the `catch()` decorator

For handled or unhandled exceptions, you can use the @logger.catch decorator or with logger.catch(): context manager. It automatically catches any exception, logs it with a full stack trace, and then re-raises it:

from loguru import logger

@logger.catch
def divide(a, b):
    return a / b

divide(1, 0)

This produces a formatted and informative traceback as before:

{
  "time": "2025-09-03T08:48:20.779242+02:00",
  "level": "ERROR",
  "message": "An error has been caught in function '<module>', process 'MainProcess' (34502), thread 'MainThread' (8437194496):",
  "exception": {
    "type": "ZeroDivisionError",
    "value": "division by zero",
    "traceback": [
      "Traceback (most recent call last):\n",
      "  File \"/Users/ayo/dev/dash0/demo/loguru-demo/.venv/lib/python3.13/site-packages/loguru/_logger.py\", line 1297, in catch_wrapper\n    return function(*args, **kwargs)\n",
      "  File \"/Users/ayo/dev/dash0/demo/loguru-demo/main.py\", line 44, in divide\n    return a / b\n           ~~^~~\n",
      "ZeroDivisionError: division by zero\n"
    ]
  }
}

The `diagnose` and `backtrace` parameters

One of Loguru's most powerful debugging features is its ability to show the values of local variables directly inside the stack trace. Enabling diagnose=True when adding a sink tells Loguru to include this extra detail, making it far easier to understand why a line of code failed.

This feature is fantastic during development but should never be used in production. With diagnose=True, sensitive information such as passwords, tokens, or personal data can easily end up in your logs. Always disable it in production by setting diagnose=False.

For production you'll also want to keep tracebacks focused on your own code. Setting backtrace=False trims away the noise from deep library internals, leaving you with a concise and readable stack trace.

logger.add(
    sys.stdout,
    level="INFO",
    serialize=True,
    diagnose=False,  # Avoid leaking sensitive data
    backtrace=False  # Show only relevant frames
)

Improving logging performance

Writing to files or sending logs over the network is I/O-heavy, and every call to log forces the calling thread to wait until the operation completes. Multiply that across many threads, and those small pauses can add up to real slowdowns.

Loguru ensures that messages stay clean and consistent even in these scenarios. All sinks are thread-safe by default so that when multiple threads log to the same resource, Loguru uses internal locks so that each message is written fully before the next begins. This prevents overlapping or corrupted log lines without any extra work on your part.

If you want to remove even that small blocking cost, the fix is to make logging asynchronous. Instead of writing directly, worker threads push log records into an in-memory queue and continue immediately. A background thread pulls from the queue and performs the slow I/O, so your main application is never delayed.

Loguru makes this pattern effortless. Just pass enqueue=True when you add a sink, and it automatically sets up the queue and background worker for you:

from loguru import logger

logger.add(
    "file.log",
    level="INFO",
    enqueue=True  # non-blocking and safe across threads/processes
)

But when your application is shutting down, there may still be messages sitting in that queue that haven't been written yet. If the program exits immediately, those messages will be lost.

That's where logger.complete() comes in. It flushes the queue by waiting until all enqueued log records are processed, and then stops the background worker cleanly.

It can be called from both synchronous or asynchronous code (with await). The typical use case is at shutdown, or right before your process exits, when you want to make sure that all logs have been written out:

from fastapi import FastAPI
from loguru import logger

app = FastAPI()

@app.on_event("shutdown")
async def shutdown_event():
    await logger.complete()

Lazy evaluation of expensive functions

Sometimes you want to log verbose or expensive details in development, but avoid paying the cost of computing them in production. Loguru's opt(lazy=True) method makes this possible by only evaluating values if the log message actually passes the sink's level filter:

def expensive_function():
    # Simulate something costly
    time.sleep(2)
    return 42

logger.opt(lazy=True).debug("Expensive result: {x}", x=lambda: expensive_function())

Here, if the configured sink is set to INFO or higher, expensive_function() is never called. The purpose of the lambda is to defer execution, so that Loguru can decide at runtime whether to actually evaluate it.

Beyond lazy evaluation, opt() also provides other per-message tweaks for handling stack traces, formatting, and context when you need them.

Writing logs to files

While the 12-Factor App methodology recommends logging to standard output, many deployments do require file-based logging. Loguru has powerful, built-in rotation and retention mechanisms that are trivial to configure:

logger.add(
    "my_app.log",
    rotation="50 MB",       # Rotates when the file reaches 50 MB
    retention="5 days",     # Keeps logs for 5 days
    compression="zip",       # Compresses old log files
    level="INFO",
    enqueue=True
)

However, it is often a better practice to offload log rotation to a dedicated system utility like logrotate so that application concerns are cleanly separated from operational concerns. This means you would simply log to a file and let logrotate handle the rest.

Disabling Loguru in tests

When running automated tests, logs are often more distracting than helpful as they clutter the output, and make failures harder to read. Most of the time you either want logging completely disabled or reduced to only critical errors.

Because all logging goes through sinks, you can control output globally by adding or removing them in your test configuration. The most straightforward approach is to remove all sinks at the start of your tests. For example:

import pytest
from loguru import logger

@pytest.fixture(autouse=True)
def disable_loguru():
    # Remove all configured sinks so nothing is printed during tests
    logger.remove()

With this fixture, every test runs with Loguru disabled by default. If you'd rather keep Loguru active but silence its output by redirecting logs to an in-memory buffer instead of the console or a file, you can use the following instead:

import io
from loguru import logger

@pytest.fixture(autouse=True)
def swallow_loguru():
    logger.remove()
    logger.add(io.StringIO())  # Logs go here but never reach stdout

Redirecting standard logging into Loguru

Frameworks, libraries, and dependencies all bring their own loggers, and nearly all of them use Python's standard logging module. The result is a flood of messages that don't match your application's formatting, don't benefit from your structured JSON output, and can quickly overwhelm you with noise unless carefully managed.

Instead of configuring dozens of different loggers by hand, you can redirect every standard logging call into your Loguru pipeline. You do this by configuring an InterceptHandler:

import logging

# [...your existing Loguru configuration]

class InterceptHandler(logging.Handler):
    def emit(self, record):
        # Get corresponding Loguru level if it exists
        try:
            level = logger.level(record.levelname).name
        except ValueError:
            level = record.levelno

        # Find caller from where originated the logged message
        frame, depth = logging.currentframe(), 2
        while frame and frame.f_code.co_filename == logging.__file__:
            frame = frame.f_back
            depth += 1

        logger.opt(depth=depth, exception=record.exc_info).log(level, record.getMessage())

# This line intercepts all logs from the standard logging module
logging.basicConfig(handlers=[InterceptHandler()], level=0, force=True)

logging.info("Standard library logging intercepted. All logs will now be handled by Loguru.")

With this handler in place, logs from the logging module will be captured by Loguru and formatted just like your application's own logs:

{
  "time": "2025-09-03T11:27:43.693500+02:00",
  "level": "INFO",
  "message": "Standard library logging intercepted. All logs will now be handled by Loguru."
}

For a comprehensive guide on switching from the standard logging module, see the Loguru migration documentation.

Bringing your Python logs into an observability pipeline

Once your Python services are emitting well structured and context-rich logs with Loguru, the next step is to move them beyond local storage, and into a centralized observability pipeline.

Centralizing your logs lets you search across services, build dashboards, and trigger alerts. Even more importantly, they can be correlated with other signals like metrics and traces to give you a complete picture of your system's health and behavior.

Modern observability platforms like Dash0 can ingest the JSON output you configure with Loguru's custom serializer. Once ingested, those logs can be filtered, aggregated, and visualized just like any other telemetry stream.

What about OpenTelemetry?

This is where we must address a significant trade-off with Loguru. Unlike the standard logging module, Loguru does not have an official, first-party integration with OpenTelemetry. This means that trace context (if available) is not propagated into your logs automatically.

However, you can build this bridge manually. The correct approach is to access the active trace context from OpenTelemetry within your application and inject it into the Loguru logger. This gives you the correlation you need for true observability.

Here is a practical example using a FastAPI middleware. This middleware will automatically grab the current trace_id and span_id and add them to the logging context for the duration of the request.

First, ensure you have the necessary OpenTelemetry packages:

pip install opentelemetry-api opentelemetry-sdk

Then update your application code as follows:

import sys
import json
import traceback
from loguru import logger
from fastapi import FastAPI, Request

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor

trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)
trace.get_tracer_provider().add_span_processor(
    SimpleSpanProcessor(ConsoleSpanExporter())
)

logger.remove()

def serialize(record):
    subset = {
        "time": record["time"].isoformat(),
        "level": record["level"].name,
        "message": record["message"],
    }
    # Merge extra fields directly into the top-level dict
    if record["extra"]:
        subset.update(record["extra"])

    if record["exception"]:
        exc = record["exception"]
        subset["exception"] = {
            "type": exc.type.__name__,
            "value": str(exc.value),
            "traceback": traceback.format_exception(exc.type, exc.value, exc.traceback),
        }

    return json.dumps(subset)

def patching(record):
    record["serialized"] = serialize(record)

logger = logger.patch(patching)


def custom_formatter(record):
    return "{serialized}\n"

logger.add(sys.stdout, level="INFO", format=custom_formatter)

app = FastAPI()

@app.middleware("http")
async def otel_logging_middleware(request: Request, call_next):
    # Start a new span for the incoming request
    with tracer.start_as_current_span("http_request") as span:
        # Get the current trace and span IDs
        span_context = span.get_span_context()
        trace_id = f'{span_context.trace_id:032x}'
        span_id = f'{span_context.span_id:016x}'

        # Add IDs to the logging context for the duration of the request
        with logger.contextualize(trace_id=trace_id, span_id=span_id):
            logger.info("Request started")
            response = await call_next(request)
            logger.info("Request finished")
            return response

@app.get("/users/{user_id}")
async def get_user(user_id: str):
    logger.info("Fetching user profile for {user_id}", user_id=user_id)
    return {"user_id": user_id, "message": "Hello from Kigali!"}

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(
        app,
        host="0.0.0.0",
        port=8000,
        reload=False,
        access_log=False,
    )

Now, when you run this application and make a request to /users/123, every log message generated during that request will automatically be enriched with the trace_id and span_id:

{"time": "2025-09-03T14:05:25.853738+02:00", "level": "INFO", "message": "Request started", "trace_id": "14a89eba2e2232303a467ff70d8dc584", "span_id": "53550a871addc2b5"}
{"time": "2025-09-03T14:05:25.854829+02:00", "level": "INFO", "message": "Fetching user profile for 12", "trace_id": "14a89eba2e2232303a467ff70d8dc584", "span_id": "53550a871addc2b5", "user_id": "12"}
{"time": "2025-09-03T14:05:25.855016+02:00", "level": "INFO", "message": "Request finished", "trace_id": "14a89eba2e2232303a467ff70d8dc584", "span_id": "53550a871addc2b5"}

While this manual setup requires more boilerplate than using a library with native OTel support, it is a robust pattern that makes your Loguru logs truly production-grade and fully integrated into a modern observability stack.

Final thoughts

Logging is often treated as an afterthought, but in production it is one of the most important windows into what your services are really doing.

Python's standard library logging module is flexible but verbose, often requiring layers of handlers, formatters, and filters before it produces something useful.

Loguru takes a different approach. By collapsing that complexity into a single logger object with the powerful add() method, it makes advanced logging accessible with just a few lines of code.

Features like structured JSON output, contextual logging, exception handling, and non-blocking sinks give you production-grade logging without the boilerplate.

Of course, Loguru isn't a silver bullet. It currently lacks first-class OpenTelemetry support, and you may still need to bridge with the standard logging module to capture logs from third-party libraries. Even so, its simplicity and flexibility make it an excellent choice for modern Python applications.

Thanks for reading!

9 Logging Best Practices You Should Know

Ayooluwa Isaiah — Tue, 14 Oct 2025 12:40:30 +0000

We continue to build increasingly complex, distributed systems, yet we often diagnose them with little more than glorified printf statements. While the practice of logging has been with us since the earliest days of computing, too many teams still treat it as an afterthought.

The consequences are all too familiar: the shocking cloud bill for debug logs that were never removed, the afternoon wasted trying to make sense of logs that say everything and nothing at the same time, and the thankless task of manually correlating events across services when your tools should have done it for you.

This guide is about fixing that. Logs aren't the whole observability story, but they can be transformed from unstructured strings scattered through a codebase into useful signals that drive real insight. The following checklist of best practices will help you do just that.

Let's begin!

1. Start with structured logging

Unstructured, string-formatted logs are an anti-pattern in modern systems. If you're still writing logs designed to be read by someone running grep on a server, you're not just behind the times, you're actively building an un-observable system.

Logs are data and must be treated as such from the moment of creation. This means every log entry must be a structured, machine-parsable object (JSON is the lingua-franca here). Every piece of information becomes a distinct key-value pair, ready to be indexed, queried, and aggregated.

Instead of emitting a blob of text like this:

[2025-07-24 07:45:10] INFO: Payment processed for user 12345 in 54ms. Request ID: abc-xyz-789

You'd output a queryable data record:

{
  "level": "info",
  "timestamp": "2025-07-24T06:45:10.123Z",
  "message": "payment processed",
  "service": "billing-api",
  "duration_ms": 54,
  "user_id": "12345",
  "trace_id": "abc-xyz-789"
}

This fundamentally changes how you interact with your logs, moving you from clumsy regex-based text search to a more powerful, query-based analysis.

This makes it trivial to answer questions like: "How many payment operations failed in the last hour?" or "Show me all logs tied to request abc-xyz-789", or even "Which logs are associated with user 12345 today?" with a simple fast query.

Most modern logging frameworks now support or default to emitting structured output out of the box, and languages are starting to treat it as a core feature rather than an optional add-on (see Go's log/slog package).

2. Establish an observability contract with OpenTelemetry

Adopting structured logging is just the first step. Without a shared standard, you're still left with the chaos of inconsistent names and semantics across services.

One service might log a user ID as "user_id": "12345", another as "userId": "12345", and yet another as "customer": { "id": "12345" }. Multiply that inconsistency across dozens of services, and the result is that observability becomes nearly impossible since everyone is speaking a different language.

To fix this, you need to establish an observability contract: a single, enforced schema for telemetry across all your services. This is where OpenTelemetry (OTel) becomes your foundation, providing a common structure (the log data model) and a common vocabulary (semantic conventions).

The good news is that you don't have to rip out your existing instrumentation. OpenTelemetry provides two clear paths (or "bridges") for bringing your logs into compliance.

For applications you control, you can integrate OTel directly with your existing logging library through an appender (or exporter). This component intercepts structured logs from your library, translates them to the OTel model in memory, and sends them directly to a collector or backend.

For legacy or third-party systems you can't change, let them continue writing to stdout or local files and have the OTel Collector ingest and transform them into the OTel model before forwarding them.

Finally, a contract is useless without enforcement. Use tools like the OpenTelemetry weaver to document and enforce a core schema of attributes that are mandatory for every service.

Then back it up with automation in your CI/CD pipeline so that builds that introduces unapproved attributes or omits required ones are automatically failed, making compliance a baked-in part of your engineering standard.

3. Enrich your logs with sufficient context

With the contract in place, you must ensure that every log is enriched with enough context to be useful on its own. A log message without context is just noise. Every log should be able to answer critical questions about its origin, scope, and intent.

You can think of context in two layers:

1. The platform context

This is the environmental and request-level context that should be attached to every single log automatically with zero developer effort. This tells you where the log came from, and what it is related to.

This is where your platform engineering shines, and OpenTelemetry provides the tools to make this automation seamless:

The OTel SDK automatically injects trace_id and span_id, linking logs to the specific request that generated them.
The OTel Collector can automatically detect and attach metadata from the host environment, such as cloud.provider, cloud.region, k8s.pod.name, and service.version using processors like the resourcedetectionprocessor and k8sattributesprocessor.

This layer is your foundation. It ensures that even the most basic log can be traced back to a specific request and service instance, in a specific region, running a specific version.

2. The event context

This is the business-level context that only your application code knows. It's the most valuable information for debugging and the hardest to get right. Examples include entity identifiers and other domain-specific attributes that explain what action was attempted and why.

You shouldn't rely on manual effort alone here. Instead, establish a pattern where a context-aware logger (available in most logging frameworks) is injected into the request lifecycle.

With this pattern, once relevant identifiers are known during request handling, they automatically flow to every downstream log line without extra effort on your part.

By layering automated platform context with systematically-injected event-level context, you'll create logs that are "born correlated", turning them from isolated messages to a rich, connected narrative of system behavior.

The OWASP logging cheat sheet provides a solid reference for useful event attributes. Just be sure to align with OTel's semantic conventions when naming them so your logs remain consistent and interoperable.

4. Use log levels as an actionable signal

Few topics in logging generate as much debate as the proper use of severity levels.

Some argue for simplicity: just use INFO and ERROR to express that your system is either doing what it's supposed to or it isn't.

That approach may look clean on paper, but in practice it collapses too much nuance. Not every anomaly is a failure, and not every failure requires paging someone. By reducing the vocabulary, you collapse important distinctions and lose the ability to separate actionable signals from supporting detail.

Adopting more granular levels like DEBUG, WARN or FATAL is far more effective. They encode meaningful distinctions: a WARN typically highlights something unusual and actionable but not urgent (like deprecation warnings), ERROR flags an actual failure, FATAL signals an unrecoverable condition that leads a process to terminate, and DEBUG captures detail for investigations without polluting production by default.

To use them well, a few principles apply:

To avoid the "boy who cried wolf" scenario, log levels should reflect severity and actionability. If no human needs to take action, it shouldn't be logged at a level that triggers an alert.
The hardest part of logging is dialing verbosity to the right level. Too much, and you balloon costs, slow down systems, and bury engineers in noise; too little, and there's nothing useful to debug with. Verbose logs have their place, but they should be scoped, short-lived, and never treated as the default.
Ensure log verbosity can be adjusted on the fly, whether for a service, a module, or even a specific user, without redeploying. This flexibility is critical when investigating live incidents.

Log levels aren't the main value of a log, but they're a powerful signal when used consistently. They help you separate the routine from the exceptional and surface low-level details only when it's needed most. Dropping down to just two levels throws away that signal for no good reason.

5. Keep sensitive data out of your logs

One of the biggest risks in logging is the accidental inclusion of Personally Identifiable Information (PII) or other sensitive data. Of all the mistakes you can make, this is the one most likely to land your company on the front page for the wrong reasons.

High-profile slip-ups from Twitter and GitHub where plaintext passwords were accidentally written to internal logs, serve as stark reminders of how serious and easy-to-make this mistake can be.

Such leaks rarely happen with malicious intent. They happen because a developer, focused on their immediate task, is unaware of the downstream security implications of their logging choices.

The only effective defense is systemic and multi-layered. You have to assume mistakes will happen and design safeguards that catch them at multiple points.

At the application level, avoid logging entire objects that may contain sensitive fields. Instead, implement logging-safe representations that exclude or mask PII by default. This means that new attributes on the object may need to be allowlisted before they are logged.

// User is a domain object that may contain sensitive fields.
type User struct {
    ID        string
    Email     string
    Password  string
    CreatedAt time.Time
}

func (u User) LogValue() slog.Value {
    // Only allowlist the fields considered safe to log.
    return slog.GroupValue(
        slog.String("id", u.ID),
        slog.Time("created_at", u.CreatedAt),
    )
}

For ad-hoc structures, most logging frameworks allow you to configure a "middleware" to automatically mask or scrub data from known sensitive fields whose key matches a blocklist before the log is written.

Finally, apply sensitive-data scrubbing in the OpenTelemetry Collector (or equivalent) to prevent anything from slipping through before logs leave your systems.

# One of the most effective techniques is using an attribute allowlist
processors:
  redaction/allowlist:
    allow_all_keys: false
    allowed_keys:
      - http.method
      - http.url
      - http.status_code

service:
  pipelines:
    logs:
      receivers: [...]
      processors: [attributes/allowlist, ...]
      exportes: [...]

Sensitive data logging mistakes are easy to make, but with layered defenses you can drastically reduce their impact. The most effective solutions are not those that blame individuals, but those that foster a culture of shared ownership over the quality and security of observability data.

6. Treat logging performance as a first-class concern

It's sometimes easy to forget that logging isn't free. Every log line consumes CPU, memory, and I/O. At small scale you'll barely notice, but at scale, it can become a real bottleneck.

To prevent logging from slowing down your services, adopt the following practices.

Choose performant libraries

Logging libraries vary widely in performance. Start by choosing modern, efficient libraries that are designed to minimize overhead. For example, in the Go ecosystem, the built-in log/slog or libraries like Zerolog and Zap are orders of magnitude faster than older options like Logrus.

Log asynchronously

Don't let your main application thread wait for a log to be written to disk or the network. Instead, log asynchronously by writing messages to a fast, in-memory buffer and letting a separate background process handle the slower I/O operations so that logging has virtually no impact on request latency.

Be mindful of hot paths

Avoid placing verbose log statements inside high-frequency loops or critical code paths. For diagnostics in these areas, use intelligent sampling or rate-limiting to gather insights without overwhelming the system with a flood of logs.

Defer expensive operations

A more subtle performance issue lies in how logging arguments are evaluated. For example, a log statement like logger.debug(f"Processing {x}") in Python evaluates the formatted string even if the DEBUG level is disabled.

The better pattern, logger.debug("Processing %s", x), defers the string formatting until it's certain the message will be emitted, saving precious cycles in critical code.

The bottom line is to treat logging code like your business logic. If you wouldn't block a request or allocate unnecessary memory in a critical path, don't let your logging library do it either.

7. Manage log volume and cost intelligently

One of the quickest ways to burn money, overwhelm your systems, and blind your engineers is to let log volume grow unchecked. At scale, ingestion and storage can run into millions per year, while noisy streams bury the very signals you need during incidents.

The answer isn't to stop logging, but to log smarter. The best place to start is at the source by cutting high-volume, low-value logs. A classic example is successful health checks from a load balancer. Those are far more useful as a metric than as endless log lines that add little value.

For even more control, some logging frameworks allow you to log continuously to an in-memory ring buffer. Under normal conditions the buffer just overwrites itself, so nothing ever leaves memory. But if the system hits an error, the buffer is dumped along with the error log to ensure that the context from the moments leading up to the failure is preserved. It keeps volume low in the happy path while still capturing rich detail when it matters most.

Another effective approach is sampling. It's rarely necessary to keep every single log, especially for routine events. You might capture all failed requests but only one out of every hundred successful ones and still get a representative view.

Sampling also helps during cascading failures: when a service starts spewing the same error on repeat, you don't need a million identical entries. A handful of representative samples tells the story just as well without overwhelming your system or ballooning your costs.

One caveat is that OpenTelemetry doesn't yet support log sampling natively. Its sampling strategies apply only to traces, so you'll need to implement log sampling through your framework or observability pipeline.

8. Make the OpenTelemetry Collector the linchpin of your pipeline

Shipping telemetry directly from your application to a backend works for small systems, but it doesn't scale. This approach leads to tight vendor coupling, performance bottlenecks, and operational headaches as you grow.

A better architecture is to put the OpenTelemetry Collector at the center of your observability pipeline. The Collector is a performant, vendor-neutral service that can ingest all your telemetry data, process them, and then route them to any number of backends.

It gives you a single place to enforce standards, redact sensitive fields, normalize formats, filter out noisy streams, or attach environment metadata automatically.

And since it handles logs, metrics, and traces together, it can outright replace log-only agents like Fluent Bit, Logstash, or Filebeat at the edge, giving you a unified pipeline instead of a patchwork of single-purpose shippers.

The payoff is flexibility. Want to send security events to one backend and application logs to another? Drop DEBUG logs in production but keep them in staging? Insert new Kubernetes metadata without developer involvement? With the Collector as the linchpin, all of these become configuration changes instead of engineering projects.

9. Choose an OpenTelemetry-native backend

You've done the hard work. You've implemented structured logging, established a schema with OpenTelemetry, enriched your logs with deep context, and built a robust pipeline with the Collector. The final step is to ensure your observability platform can capitalize on this investment.

This is where choosing an OpenTelemetry-native platform becomes critical. An OTel-native platform isn't just a backend that can accept OTLP data; its one whose entire data model is built around the OpenTelemetry standard.

This means it inherently understands the intrinsic relationships between your signals and treats semantic conventions as first-class citizens, not just another set of tags.

It knows that db.query.text isn't just a string but database query, and can parse the statement to identify the operation, highlight slow queries and provide pre-built dashboards for database performance. These are the principles Dash0 is built on.

By pairing your high-quality instrumentation with a platform that speaks the same native language, you ensure the payoff for all your effort is not just better data, but faster debugging, clearer insights, and more reliable systems.

Final thoughts

When done well, logging shifts from being a costly liability to a force multiplier for cutting incident response times, reducing operational overhead, and making your systems more reliable.

The next time something breaks in production (and it will) the quality of your logs will determine whether you're guessing in the dark or diagnosing with confidence.

Thanks for reading!

Forem: Dash0

OpenTelemetry Filelog Receiver: A Guide to Ingesting Log Files

How the Filelog receiver works

Quick Start: tailing a log file

Filtering and managing log files

Excluding files by modification age

Parsing unstructured text with regular expressions

Handling multiple log formats

Handling stack traces and multiline logs

Parsing metadata from file headers

How to avoid lost or duplicate logs

Handling log delivery failures gracefully

Deleting log files after processing

Handling log rotation seamlessly

Reading compressed files

Performance tuning for high-volume environments

Enforcing log file order

Filelog receiver tips and best practices

Log files are not being watched

Files are watched but no log lines are read

Regular expression doesn't match log lines

Logs are duplicated after restart

Final thoughts

Leveling Up Your Python Logs with Structlog

Understanding the Structlog philosophy

Examining the default configuration

The production configuration: machine-readable JSON

How log levels work in Structlog

Filtering and dropping events

Filtering by call site information

Writing logs to files

Mastering contextual logging

Reliable context propagation in web apps

Adding global context

Capturing Python errors and exceptions

Integrating Structlog with OpenTelemetry

Final thoughts

Mastering Docker Logs: A Comprehensive Tutorial

Quick start: the docker logs command reference

Understanding how Docker logging works

Where Docker stores container logs

Viewing container logs with the docker logs command

Filtering logs by time (--since and --until)

Following and tailing Docker container logs

Filtering Docker logs with grep

Saving Docker logs to a file

Managing logs in Docker Compose

Viewing logs for a single service

Viewing logs for all services

Tailing and filtering

Inspecting Docker container logs with a GUI

1. Docker Desktop logs

2. Dozzle

Choosing a Docker logging driver

Setting a global logging driver

Overriding the logging driver per container

Understanding Docker's log delivery mode

1. Blocking mode

2. Non-blocking mode

Centralizing Docker logs with OpenTelemetry

Troubleshooting common issues with Docker container logs

1. docker logs shows no output

2. Logging driver does not support reading

Some best practices for Docker container logging

1. Write application logs to stdout and stderr

2. Use non-blocking mode for network-based logging drivers

3. Include container metadata in logs

4. Use the local or json-file driver with log rotation

Final thoughts

From Zero to Pipeline: An OpenTelemetry Collector Guide

The simplest possible pipeline

Receivers

Exporters

Service

Understanding the debug exporter output

ResourceLogs and Resource attributes

ScopeLogs

LogRecord

Processing and transforming telemetry

Batching telemetry for efficiency

Quick start: the `docker logs` command reference

Viewing container logs with the `docker logs` command

Filtering logs by time (`--since` and `--until`)

Filtering Docker logs with `grep`

1. `docker logs` shows no output

4. Use the `local` or `json-file` driver with log rotation

Understanding the `debug` exporter output

Using the `catch()` decorator

The `diagnose` and `backtrace` parameters