Forem: Adnan Rahić

Custom OpenTelemetry Collectors: Build, Run, and Manage at Scale

Adnan Rahić — Thu, 11 Sep 2025 11:47:15 +0000

I tried thinking back to when the last time I read an actual tutorial that did not include a bunch of em (—) dashes, semicolons, normal dashes, and an unnervingly large quantity of the phrases like “XYZ-thing Alert 🚨” and “Exciting News!”.

Well, hold on to your suspenders folks, here we go again. Part 2 is up and it’s a controversial one. 👇

Have you tried building your own, custom, OpenTelemetry Collector distribution? Did you like it? 😂

I bet you’re NOT smiling. Let alone laughing out loud at that statement like I am right now! I tried, and boy did I have a hard time.

I do have a nice solution to the current norm of building custom OpenTelemetry Collectors, if you have the patience to stick around and read for the next 4 minutes. I’ll do my best to be respectful of your time and cut it down to the bare bones you need to be successful yourself. 🤝

As a normal human I find this hard…

There are a few missing steps in the existing resources and docs around using the OpenTelemetry Collector Builder (OCB). I felt like stumbling through a dark forest and barely making it out the other side. The OCB is an amazing tool for Go developers. That’s the kicker though. A lot of us are not seasoned engineers, let alone experienced Go developers.

That’s why I wanted to write this tutorial. I’ll show you a hands-on guide with the open-source OpenTelemetry Distribution Builder (ODB) from Bindplane. You’ll learn how to build a custom, OpAMP-enabled collector using a manifest.yaml file and GitHub Actions.

Building custom OpenTelemetry collectors is a real need

Custom OpenTelemetry Collectors are no longer a niche thing. As more devs run collectors in intricate Kubernetes environments, or in containers in general, even in the edge, trimming down the binary is becoming standard practice.

Why build a custom collector?

The upstream OpenTelemetry Collector Contrib ships with a lot of components. That’s great for getting started, but in production you don’t need all of them. Simply put, more components equals bigger binaries and more attack surface.

A custom built collector solves that. You define exactly what is needed:

Only receivers, processors, exporters, and extensions you use
Minimal footprint
No unnecessary dependencies

The OpenTelemetry Distribution Builder (ODB)

ODB is Bindplane’s open-source builder for creating custom collectors.

You feed it a manifest.yaml and it gives you binaries and packages for every platform you need.

What you get:

Multi-platform builds: Linux, Windows, macOS, AMD64, ARM64
Multiple formats: .tar.gz, .zip, .deb, .rpm
No Go coding, no manual dependency resolution
OpAMP support (Bindplane-compatible out-of-the-box)

Step 1 — Create a GitHub repo

Start by creating a blank GitHub repo to store the manifest.yaml and run GitHub Action Workflows.

Next, create a new repo in your local env.

echo "# otel-distro-builder-github-action" >> README.md
git init
git add README.md
git commit -m "first commit"
git branch -M main
git remote add origin git@github.com:<YOUR_ACCOUNT_NAME>/otel-distro-builder-github-action.git
git push -u origin main

The repo is now ready to add a manifest.yaml.

Step 2 — Write a `manifest.yaml`

The manifest.yaml is where you define what goes in your collector.

Here’s a suggested example that I’ve vetted with my colleagues at Bindplane that contribute to the OpenTelemetry project. It’s minimal, but still includes quality-of-life modules like OpAMP support and common processors.

dist:
  module: github.com/<YOUR-USERNAME>/my-custom-opentelemetry-distro
  name: my-custom-opentelemetry-distro
  description: Custom-built OpenTelemetry Collector.
  output_path: ./_build
  version: v0.0.1
conf_resolver:
  default_uri_scheme: "env"
receivers:
  - gomod: go.opentelemetry.io/collector/receiver/nopreceiver v0.128.0
  - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver v0.128.0
  - gomod: github.com/observiq/bindplane-otel-collector/receiver/telemetrygeneratorreceiver v1.79.0
exporters:
  - gomod: go.opentelemetry.io/collector/exporter/debugexporter v0.128.0
  - gomod: go.opentelemetry.io/collector/exporter/nopexporter v0.128.0
  - gomod: go.opentelemetry.io/collector/exporter/otlpexporter v0.128.0
  - gomod: go.opentelemetry.io/collector/exporter/otlphttpexporter v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter v0.128.0
  - gomod: github.com/observiq/bindplane-otel-collector/exporter/googlecloudstorageexporter v1.79.0
extensions:
  - gomod: go.opentelemetry.io/collector/extension/zpagesextension v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/ackextension v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/asapauthextension v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/awsproxy v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/basicauthextension v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/bearertokenauthextension v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/encoding/jaegerencodingextension v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/encoding/otlpencodingextension v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/encoding/zipkinencodingextension v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/headerssetterextension v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/healthcheckextension v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/httpforwarderextension v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/jaegerremotesampling v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/oauth2clientauthextension v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/observer/dockerobserver v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/observer/ecsobserver v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/observer/ecstaskobserver v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/observer/hostobserver v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/observer/k8sobserver v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/oidcauthextension v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/opampextension v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/pprofextension v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/sigv4authextension v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage/filestorage v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage/dbstorage v0.128.0
  - gomod: github.com/observiq/bindplane-otel-collector/extension/bindplaneextension v1.79.0
processors:
  - gomod: go.opentelemetry.io/collector/processor/batchprocessor v0.128.0
  - gomod: go.opentelemetry.io/collector/processor/memorylimiterprocessor v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/attributesprocessor v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/cumulativetodeltaprocessor v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/deltatorateprocessor v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/filterprocessor v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/groupbyattrsprocessor v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/groupbytraceprocessor v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/k8sattributesprocessor v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/logdedupprocessor v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/metricsgenerationprocessor v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/metricstransformprocessor v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/probabilisticsamplerprocessor v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/redactionprocessor v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/remotetapprocessor v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourceprocessor v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/routingprocessor v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/spanprocessor v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/sumologicprocessor v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/transformprocessor v0.128.0
  - gomod: github.com/observiq/bindplane-otel-collector/processor/datapointcountprocessor v1.79.0
  - gomod: github.com/observiq/bindplane-otel-collector/processor/logcountprocessor v1.79.0
  - gomod: github.com/observiq/bindplane-otel-collector/processor/lookupprocessor v1.79.0
  - gomod: github.com/observiq/bindplane-otel-collector/processor/maskprocessor v1.79.0
  - gomod: github.com/observiq/bindplane-otel-collector/processor/metricextractprocessor v1.79.0
  - gomod: github.com/observiq/bindplane-otel-collector/processor/metricstatsprocessor v1.79.0
  - gomod: github.com/observiq/bindplane-otel-collector/processor/removeemptyvaluesprocessor v1.79.0
  - gomod: github.com/observiq/bindplane-otel-collector/processor/resourceattributetransposerprocessor v1.79.0
  - gomod: github.com/observiq/bindplane-otel-collector/processor/samplingprocessor v1.79.0
  - gomod: github.com/observiq/bindplane-otel-collector/processor/snapshotprocessor v1.79.0
  - gomod: github.com/observiq/bindplane-otel-collector/processor/spancountprocessor v1.79.0
  - gomod: github.com/observiq/bindplane-otel-collector/processor/throughputmeasurementprocessor v1.79.0
  - gomod: github.com/observiq/bindplane-otel-collector/processor/topologyprocessor v1.79.0
  - gomod: github.com/observiq/bindplane-otel-collector/processor/unrollprocessor v1.79.0
connectors:
  - gomod: go.opentelemetry.io/collector/connector/forwardconnector v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/connector/countconnector v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/connector/datadogconnector v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/connector/exceptionsconnector v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/connector/failoverconnector v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/connector/grafanacloudconnector v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/connector/roundrobinconnector v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/connector/routingconnector v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/connector/servicegraphconnector v0.128.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/connector/spanmetricsconnector v0.128.0
providers:
  - gomod: go.opentelemetry.io/collector/confmap/provider/envprovider v1.34.0
  - gomod: go.opentelemetry.io/collector/confmap/provider/fileprovider v1.34.0
  - gomod: go.opentelemetry.io/collector/confmap/provider/httpprovider v1.34.0
  - gomod: go.opentelemetry.io/collector/confmap/provider/httpsprovider v1.34.0
  - gomod: go.opentelemetry.io/collector/confmap/provider/yamlprovider v1.34.0
# When adding a replace, add a comment before it to document why it's needed and when it can be removed
replaces:
  # See https://github.com/google/gnostic/issues/262
  - github.com/googleapis/gnostic v0.5.6 => github.com/googleapis/gnostic v0.5.5
  # See https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/12322#issuecomment-1185029670
  - github.com/docker/go-connections v0.4.1-0.20210727194412-58542c764a11 => github.com/docker/go-connections v0.4.0
  # see https://github.com/mattn/go-ieproxy/issues/45
  - github.com/mattn/go-ieproxy => github.com/mattn/go-ieproxy v0.0.1
  # see https://github.com/openshift/api/pull/1515
  - github.com/openshift/api => github.com/openshift/api v0.0.0-20230726162818-81f778f3b3ec
  - github.com/observiq/bindplane-otel-collector/internal/version => github.com/observiq/bindplane-otel-collector/internal/version v0.0.0-20250306153219-6fe3f849c29f

💡 The opampextension is the key — it’s what lets Bindplane discover and manage your collector.

Step 3 — Automate the build with GitHub Actions

Use the **OpenTelemetry Distribution Builder GitHub Action** to do the heavy lifting.

Here’s a .github/workflows/multi.yaml you can drop in:

name: Matrixed OpenTelemetry Distribution Build

on:
  push:
    tags:
      - "v*" # Runs when a version tag is pushed (e.g., v1.0.0)
  workflow_dispatch: # Enables manual triggering from the GitHub UI

permissions:
  contents: write # This is required for creating/modifying releases

jobs:
  build:
    # Configure build matrix to run multiple platform builds in parallel
    strategy:
      matrix:
        # Define the platforms we want to build for
        platform: [linux/amd64, linux/arm64, darwin/arm64, windows/amd64]
        # Map platform identifiers to simpler names for artifact handling
        include:
          - platform: linux/amd64
            os: linux
            arch: amd64
            artifact_name: linux-amd64
          - platform: linux/arm64
            os: linux
            arch: arm64
            artifact_name: linux-arm64
          - platform: darwin/arm64
            os: darwin
            arch: arm64
            artifact_name: darwin-arm64

          - platform: windows/amd64
            os: windows
            arch: amd64
            artifact_name: windows-amd64
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # Build the OpenTelemetry distribution for each platform
      - name: Build and Package
        uses: observiq/otel-distro-builder@main
        with:
          os: ${{ matrix.os }}
          arch: ${{ matrix.arch }}

      # Upload platform-specific artifacts with a unique name
      # These artifacts are available for download in the GitHub Actions UI
      - name: Upload Artifacts
        uses: actions/upload-artifact@v4
        with:
          name: distribution-artifacts-${{ matrix.artifact_name }}
          path: |
            ${{ startsWith(matrix.platform, 'linux') && format('{0}/artifacts/*.deb', github.workspace) || '' }}
            ${{ startsWith(matrix.platform, 'linux') && format('{0}/artifacts/*.rpm', github.workspace) || '' }}
            ${{ startsWith(matrix.platform, 'linux') && format('{0}/artifacts/*.apk', github.workspace) || '' }}
            ${{ startsWith(matrix.platform, 'windows') && format('{0}/artifacts/*.zip', github.workspace) || '' }}
            ${{ (startsWith(matrix.platform, 'linux') || startsWith(matrix.platform, 'darwin')) && format('{0}/artifacts/*.tar.gz', github.workspace) || '' }}
            ${{ format('{0}/artifacts/*.sbom.json', github.workspace) || '' }}
            ${{ format('{0}/artifacts/*_checksums.txt', github.workspace) || '' }}
          retention-days: 5 # Artifacts are kept for 5 days then automatically deleted

  # Create a single release containing all platform artifacts
  release:
    needs: build # Wait for all platform builds to complete
    runs-on: ubuntu-latest
    permissions:
      contents: write # Required permission for creating releases
    steps:
      # Download all platform-specific artifacts into a single directory
      - name: Download All Artifacts
        uses: actions/download-artifact@v4
        with:
          path: all-artifacts
          pattern: distribution-artifacts-* # Match all our platform-specific artifacts
          merge-multiple: true # Combine all artifacts into a single directory

      # Create a GitHub Release and attach all platform artifacts
      # This makes all platform builds available for download from the Releases page
      - name: Create Release
        uses: softprops/action-gh-release@v2
        with:
          files: all-artifacts/**/*

Every new version release will:

Build your custom collector for all platforms
Package it into .tar.gz, .zip, .deb, and .rpm
Store them as GitHub Actions artifacts

You should have two files created and ready to add to Git.

git status

[Output]
On branch main
Your branch is up to date with 'origin/main'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
    .github/
    manifest.yaml

Commit these changes and push them to your repo.

git add .github manifest.yaml
git commit -m "add a manifest and workflow"
git push origin main

Create a new release. Make sure to use the same release tag as you specified in your manifest.yaml. In the sample manifest.yaml above I used v0.0.1 which means I need to set the same tag in the release.

Once the release is created, you’ll see it’s initially empty.

Opening the Actions will show the build running.

Give it about 5 minutes to complete. Go get a coffee. ☕

Once the builds are done, you’ll see artifacts saved and added to the release.

Step 4 — Download and run your collector

Let me show you how to run your custom collector in a Linux VM. Grab an artifact from the Release and extract it.

wget https://github.com/adnanrahic/otel-distro-builder-github-action/releases/download/v0.0.1/my-custom-opentelemetry-distro_otelcol_v0.0.1_linux_amd64.tar.gz

tar -xvzf my-custom-opentelemetry-distro_otelcol_v0.0.1_linux_amd64.tar.gz

You’ll also get a skeleton collector_config.yaml for the custom collector bundled in the tar as well.

ls -l

[Condensed Output]
-rw-r--r-- collector_config.yaml
-rwxr-xr-x my-custom-opentelemetry-distro
-rw-r--r-- my-custom-opentelemetry-distro_otelcol_v0.0.1_linux_amd64.tar.gz
drwxr-xr-x service

Let’s edit it slightly, and also add Bindplane’s OpAMP configuration by following this guide. Paste this into the config file.

# To limit exposure to denial of service attacks, change the host in endpoints below from 0.0.0.0 to a specific network interface.
# See https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks

extensions:
  health_check:
  pprof:
    endpoint: 0.0.0.0:1777
  zpages:
    endpoint: 0.0.0.0:55679

    # Add an OpAMP connection to Bindplane.
    # Use the credentials from your account, and
    # follow the guide from the screenshot below.
  opamp:
    instance_uid: 01K42T4MGFFDZMXBY7C5C2APX9 # Generated ULID
    capabilities:
      reports_effective_config: true
    server:
      ws:
          # Bindplane Cloud OpAMP Endpoint
        endpoint: wss://app.bindplane.com/v1/opamp
        headers:
          Authorization: Secret-Key <YOUR_SECRET_KEY> 
          X-Bindplane-Labels: <YOUR_LABEL=YOUR_VALUE>
        tls:
          insecure: false

receivers:
  telemetrygeneratorreceiver/logs:
    generators:
      - additional_config:
          body: 'foo: bar'
          severity: 9
        type: logs
    payloads_per_second: 1

  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:

exporters:
  debug:
    verbosity: detailed
  nop: null

service:

  pipelines:

    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, nop]

    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, nop]

    logs:
      receivers: [telemetrygeneratorreceiver/logs, otlp]
      processors: [batch]
      exporters: [debug, nop]

  extensions: [health_check, pprof, zpages, opamp]

**Note:* If you’re on the Growth or Enterprise plans of Bindplane, and want to connect the collector Bindplane, use the secret key after the -s and the labels after the -k. The collector will work as expected with or without connecting to Bindplane. But, you will not get the management capabilities and benefits of OpAMP.*

Now, go ahead and run the collector binary.

./my-custom-opentelemetry-distro --config collector_config.yaml

You’ll see logs like this show as the terminal output confirming the telemetrygeneratorreceiver is creating some dummy logs to validate your config is working.

2025-09-02T08:44:44.255Z    info    service@v0.128.0/service.go:282 Everything is ready. Begin running and processing data. {"resource": {"service.instance.id": "8c4d008a-700d-49d6-b6ed-41cd1e03cf18", "service.name": "my-custom-opentelemetry-distro", "service.version": "v0.0.1"}}

2025-09-02T08:44:44.454Z    info    Logs    {"resource": {"service.instance.id": "8c4d008a-700d-49d6-b6ed-41cd1e03cf18", "service.name": "my-custom-opentelemetry-distro", "service.version": "v0.0.1"}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "resource logs": 1, "log records": 1}

2025-09-02T08:44:44.455Z    info    ResourceLog #0
Resource SchemaURL: 
ScopeLogs #0
ScopeLogs SchemaURL: 
InstrumentationScope  
LogRecord #0
ObservedTimestamp: 2025-09-02 08:44:44.253813518 +0000 UTC
Timestamp: 2025-09-02 08:44:44.253813518 +0000 UTC
SeverityText: 
SeverityNumber: Info(9)
Body: Str(foo: bar)
Trace ID: 
Span ID: 
Flags: 0

Within seconds, your custom collector will appear in Bindplane’s Agents list.

You can open the collector config as well.

Bindplane is picking up the collector metadata and the config you added as well. The OpAMP Extension currently doesn't support remote configuration, which means you cannot modify the Collector configuration through the Bindplane UI. You can still view the current Collector configuration as YAML on the Collector page, but the "Choose Another Configuration" button will not be available.

Let me walk you through adding your custom collector to Bindplane and enabling remote configurations.

Step 5 — Remotely manage your custom collector with OpAMP

You can enable remote collector management by adding your custom collector as an Agent Type in Bindplane as outlined in this docs guide. Let me walk you through it step-by-step. 🚶‍♀️

1. Install the Bindplane CLI

The Bindplane CLI lets you manage Bindplane resources including Agent Types. Follow the OS-specific installation steps here.

2. Create an API Key & set a `default` profile

Create an API Key to access resources in Bindplane with the CLI. Follow the steps here.

3. Create an Agent Type in Bindplane

In Bindplane, an Agent Type represents an OpenTelemetry Collector Distribution. For example, the BDOT v1 and v2 collectors are both Agent Types.

apiVersion: bindplane.observiq.com/v1
kind: AgentType
metadata:
  name: my-custom-opentelemetry-distro
  displayName: My Custom OpenTelemetry Distro
  description: My custom OpenTelemetry collector distro.
spec:
  repositoryLink: https://github.com/<YOUR_GITHUB_USERNAME>/<YOUR_REPO_NAME>
  platformArchSet:
    - platform: darwin
      arch: arm64
    - platform: linux
      arch: amd64
    - platform: linux
      arch: arm64
    - platform: windows
      arch: amd64

This is where you need to be careful. The repositoryLink needs to match the link of the repo where you ran the GitHub action. In my example it was:

https://github.com/adnanrahic/otel-distro-builder-github-action

The metadata.name value also needs to match the value of dist.name in your manifest.yaml.

Apply the custom Agent Type.

bindplane apply -f /path/to/agent/type/file.yaml

Finally, sync the Agent Type to load it into Bindplane.

bindplane sync agent-versions --agent-type my-custom-opentelemetry-distro --version v0.0.1

Note that the version matches the release.

5. Install your custom collector from the Bindplane UI

Now since you’ve added a custom Agent Type and synced a version, you can choose to install it from the Bindplane UI.

Since you built it for Mac, Windows, and Linux, you’ll see all three options when selecting platform.

I want to install it in my Linux VM, so I’ll select Linux and click next.

I’ll get this generic install single-command. Running this in my VM will start my custom collector and hook it up to Bindplane via OpAMP.

sudo sh -c "$(curl -fsSlL 'https://raw.githubusercontent.com/observIQ/bindplane-otel-collector/refs/heads/main/scripts/generic-install/install_unix.sh')" install_unix.sh -d 'my-custom-opentelemetry-distro' -u 'https://github.com/adnanrahic/otel-distro-builder-github-action' -e 'wss://app.bindplane.com/v1/opamp' -s '01J06XSD7FVM3CHCQA3823AC2X' -v '0.0.1' -l 'install_id=937a3445-8962-441a-90f0-ee120c67edb7'

[Output]
Using repository URL: https://github.com/adnanrahic/otel-distro-builder-github-action
Auto-detected package type: deb
Downloading: https://github.com/adnanrahic/otel-distro-builder-github-action/releases/download/v0.0.1/my-custom-opentelemetry-distro_v0.0.1_linux_amd64.deb
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 46.9M  100 46.9M    0     0  38.4M      0  0:00:01  0:00:01 --:--:-- 98.4M
Selecting previously unselected package my-custom-opentelemetry-distro.
(Reading database ... 76875 files and directories currently installed.)
Preparing to unpack .../my-custom-opentelemetry-distro_0.0.1.deb ...
Unpacking my-custom-opentelemetry-distro (0.0.1) ...
Setting up my-custom-opentelemetry-distro (0.0.1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/my-custom-opentelemetry-distro.service → /lib/systemd/system/my-custom-opentelemetry-distro.service.
Creating supervisor config...
Managing service state...
Starting my-custom-opentelemetry-distro service...
Service is running
Installation complete!
Installation directory: /opt/my-custom-opentelemetry-distro
Supervisor config: /opt/my-custom-opentelemetry-distro/supervisor_config.yaml

This will start the collector as a systemd service.

sudo systemctl status my-custom-opentelemetry-distro

[Output]
● my-custom-opentelemetry-distro.service - An OpenTelemetry Collector service named 'my-custom-opentelemetry-distro'.
     Loaded: loaded (/lib/systemd/system/my-custom-opentelemetry-distro.service; enabled; preset: enabled)
     Active: active (running) since Tue 2025-09-02 12:36:56 UTC; 2min 51s ago
   Main PID: 17005 (supervisor)
      Tasks: 7 (limit: 4681)
     Memory: 8.5M
        CPU: 249ms
     CGroup: /system.slice/my-custom-opentelemetry-distro.service
             └─17005 /opt/my-custom-opentelemetry-distro/supervisor --config=/opt/my-custom-opentelemetry-distro/supervisor_config.yaml

Sep 02 12:36:56 my-custom-opentelemetry-distro systemd[1]: Started my-custom-opentelemetry-distro.service - An OpenTelemetry Collector service named 'my-opentelemetry-distro.service - An OpenTelemetry Collector service named 'my-custom-opentelemetry-distro'.

Because the OpAMP connection works via websockets, it’ll update the UI right away and show you the collector running.

6. Configure and manage your custom collector from the Bindplane UI

You can now create a configuration for the collector and remote push it down.

Click the Create Configuration button to create and manage a config in Bindplane and apply it remotely.

Give it a name, select the Agent Type for your custom collector, and select the platform where your custom collector is running. For my example, it’s Linux. Add a Telemetry Generator source.

And, a Dev Null destination.

This will finalize your config creation. You still need to connect it to your custom collector.

Click the Add Agents button. Select your custom collector, and hit save.

Now, you can start a rollout to apply the config remotely.

What’s awesome here is that Bindplane reads the collector’s capabilities directly from the build, so you only see the components you actually included in the manifest.

Let me show you by adding a new source.

You’ll see which sources are incompatible are which you can use. This is a huge quality-of-life improvement and convenience across your entire team when creating and managing collector configs.

Step 6 — Iterate with confidence

With this setup you can:

Update the manifest.yaml to add or remove modules
Create a new release
GitHub Actions builds a new version
Deploy or upgrade in your environment
Bindplane instantly manages the updated agent

You now have a BYOC (Bring Your Own Collector) workflow — fully automated, versioned, and controlled by you.

Why this works so well

Feature	Value
OpAMP out-of-the-box	Full Bindplane remote config and monitoring
Declarative manifest	No Go code, no manual dependency resolution
GitHub-native CI/CD	Push → Build → Package → Manage
Multi-platform packaging	Build once, run anywhere
UI-aware Bindplane integration	Only shows supported components

Future goals

Moving forward I would love to abstract away the manifest.yaml as well. In an ideal world I would want to give the OpenTelemetry Distro Builder a sample collector config file. It should then be able to create a manifest.yaml from my config. This process would abstract away everything except for the specific receivers, exporters, extensions, and processors I really need.

More on this by the end of the year. 😉

Final thoughts

Custom collectors aren’t just for power users anymore. With ODB and GitHub Actions, you can build exactly what you need, package it for every platform, and manage it at scale with Bindplane. All without touching a Go compiler. 🔥

It’s clean. It’s fast. And it’s production-ready.

How to Build Resilient Telemetry Pipelines with the OpenTelemetry Collector: High Availability and Gateway Architecture

Adnan Rahić — Fri, 25 Jul 2025 12:35:38 +0000

Do you remember when humans used to write step-by-step tutorials?

Let’s bring that back. Today you’ll learn how to configure high availability for the OpenTelemetry Collector so you don’t lose telemetry during node failures, rolling upgrades, or traffic spikes. The guide covers both Docker and Kubernetes samples with hands-on demos of configs.

But first, let’s lay some groundwork.

How to define High Availability (HA) with the OpenTelemetry Collector?

You want to ensure telemetry collection and processing works even if individual Collector instances fail. It’s outlined in three main points:

Avoid data loss when exporting to a dead observability backend.
Ensure telemetry continuity during rolling updates or infrastructure failures.
Enable horizontal scalability for load-balancing traces, logs, and metrics.

To enable high availability it’s recommended that you use the Agent-Gateway deployment pattern. This means:

Agent Collectors run on every host, container, or node.
Gateway Collectors are centralized, scalable back-end services receiving telemetry from Agent Collectors.
Each layer can be scaled independently and horizontally.

Please note, an Agent Collector and Gateway Collector is essentially the same binary. They’re completely identical. The ONLY difference is WHERE it is running. Think of it this way.

An Agent Collector runs close to the workload–in the context of Kubernetes it could be a sidecar, or a deployment for every namespace–or for Docker, a service alongside your app in the docker-compose.yaml. This would tend to mean the dev team will own this instance of the Collector.
A Gateway Collector is a central (standalone) operation of the collector–think a standalone Collector in a specific namespace or even a dedicated Kubernetes cluster–typically owned by the platform team. This is the final step of the telemetry pipeline letting the platform team enforce policies like filtering logs, sampling traces, dropping metrics, before sending it to an observability backend.

Here’s an awesome explanation on StackOverflow. Yes, it’s still a thing. No, not everything is explained by AI. 😂

To satisfy all high availability I’ll walk you through how to configure:

Multiple Collector Instances. Each instance is capable of handling the full workload with redundant storage for temporary data buffering.
A Load Balancer. It’ll distribute incoming telemetry data and maintain consistent routing. Load balancers also support automatic failover if a collector becomes unavailable.
Shared Storage. Persistent storage for collector state and configuration management.

Now it’s time to get our hands dirty with some hands-on coding.

Configure Agent-Gateway High Availability (HA) with the OpenTelemetry Collector

Let me first explain this concept by using Docker and visualize it with Bindplane. This architecture is transferable and usable for any type of Linux or Windows VM setup as well. More about Kubernetes further below.

There are three options you can use. Either using a load balancer like Nginx or Traefik. Or, using the loadbalancing exporter that’s available in the Collector. Finally, if you’re fully committed to a containerized environment, use native load balancing in Kubernetes.

Nginx Load Balancer

The Nginx option is the simpler, out-of-the-box solution.

I’ll set up the architecture with:

Three Gateway Collectors in parallel
One Nginx load balancer
One Agent Collector configured to generate telemetry (app simulation)

This structure is the bare-bones minimum you’ll end up using. Note that you'll end up using three separate services for the gateway collectors. The reason behind this is that each collector needs to have its own separate file_storage path to store data in the persistent queue. In Docker, this means you need to make sure each container gets a unique volume. Let me explain how that works.

Copy the content below into a docker-compose.yaml.

version: '3.8'

volumes:
  gw1-storage:      # persistent queue for gateway-1
  gw2-storage:      # persistent queue for gateway-2
  gw3-storage:      # persistent queue for gateway-3
  telgen-storage:   # persistent queue for telemetry generator
  external-gw-storage: # persistent queue for external gateway

services:
  # ────────────── GATEWAYS (3×) ──────────────
  gw1:
    image: ghcr.io/observiq/bindplane-agent:1.79.2
    container_name: gw1
    hostname: gw1
    command: ["--config=/etc/otel/config/config.yaml"]
    volumes:
      - ./config:/etc/otel/config
      - gw1-storage:/etc/otel/storage  # 60 GiB+ queue
    environment:
      OPAMP_ENDPOINT: "wss://app.bindplane.com/v1/opamp"   # point to your Bindplane server
      OPAMP_SECRET_KEY: "<secret>"
      OPAMP_LABELS: ephemeral=true
      MANAGER_YAML_PATH: /etc/otel/config/gw1-manager.yaml
      CONFIG_YAML_PATH: /etc/otel/config/config.yaml
      LOGGING_YAML_PATH: /etc/otel/config/logging.yaml

  gw2:
    image: ghcr.io/observiq/bindplane-agent:1.79.2
    container_name: gw2
    hostname: gw2
    command: ["--config=/etc/otel/config/config.yaml"]
    volumes:
      - ./config:/etc/otel/config
      - gw2-storage:/etc/otel/storage  # 60 GiB+ queue
    environment:
      OPAMP_ENDPOINT: "wss://app.bindplane.com/v1/opamp"   # point to your Bindplane server
      OPAMP_SECRET_KEY: "<secret>"
      OPAMP_LABELS: ephemeral=true
      MANAGER_YAML_PATH: /etc/otel/config/gw2-manager.yaml
      CONFIG_YAML_PATH: /etc/otel/config/config.yaml
      LOGGING_YAML_PATH: /etc/otel/config/logging.yaml
    
  gw3:
    image: ghcr.io/observiq/bindplane-agent:1.79.2
    container_name: gw3
    hostname: gw3
    command: ["--config=/etc/otel/config/config.yaml"]
    volumes:
      - ./config:/etc/otel/config
      - gw3-storage:/etc/otel/storage  # 60 GiB+ queue
    environment:
      OPAMP_ENDPOINT: "wss://app.bindplane.com/v1/opamp"   # point to your Bindplane server
      OPAMP_SECRET_KEY: "<secret>"
      OPAMP_LABELS: ephemeral=true
      MANAGER_YAML_PATH: /etc/otel/config/gw3-manager.yaml
      CONFIG_YAML_PATH: /etc/otel/config/config.yaml
      LOGGING_YAML_PATH: /etc/otel/config/logging.yaml

  # ────────────── OTLP LOAD-BALANCER ──────────────
  otlp-lb:
    image: nginx:1.25-alpine
    volumes:
      - ./nginx-otlp.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP/JSON
    depends_on: [gw1, gw2, gw3]

  # ────────────── TELEMETRY GENERATOR ──────────────
  telgen:
    image: ghcr.io/observiq/bindplane-agent:1.79.2
    container_name: telgen
    hostname: telgen
    command: ["--config=/etc/otel/config/config.yaml"]
    volumes:
      - ./config:/etc/otel/config
      - telgen-storage:/etc/otel/storage  # 60 GiB+ queue
    environment:
      OPAMP_ENDPOINT: "wss://app.bindplane.com/v1/opamp"   # point to your Bindplane server
      OPAMP_SECRET_KEY: "<secret>"
      OPAMP_LABELS: ephemeral=true
      MANAGER_YAML_PATH: /etc/otel/config/telgen-manager.yaml
      CONFIG_YAML_PATH: /etc/otel/config/config.yaml
      LOGGING_YAML_PATH: /etc/otel/config/logging.yaml

  # ────────────── EXTERNAL GATEWAY ──────────────
  external-gw:
    image: ghcr.io/observiq/bindplane-agent:1.79.2
    container_name: external-gw
    hostname: external-gw
    command: ["--config=/etc/otel/config/external-gw-config.yaml"]
    volumes:
      - ./config:/etc/otel/config
      - external-gw-storage:/etc/otel/storage  # 60 GiB+ queue
    environment:
      OPAMP_ENDPOINT: "wss://app.bindplane.com/v1/opamp"   # point to your Bindplane server
      OPAMP_SECRET_KEY: "<secret>"
      OPAMP_LABELS: ephemeral=true
      MANAGER_YAML_PATH: /etc/otel/config/external-gw-manager.yaml
      CONFIG_YAML_PATH: /etc/otel/config/external-gw-config.yaml
      LOGGING_YAML_PATH: /etc/otel/config/logging.yaml

Open your Bindplane instance and click the Install Agent button.

Set the platform to Linux, since I’m demoing this with Docker, and hit next.

This screen now shows the environment variables you'll need to replace in the docker-compose.yaml.

Go ahead and replace the OPAMP_SECRET_KEYwith your own secret key from Bindplane. If you’re using a self-hosted instance of Bindplane, replace your OPAMP_ENDPOINTas well. Use the values after -e and -s which represent the endpoint and secret.

Create a nginx-otlp.conf file for the load balancer.

worker_processes auto;
events { worker_connections 1024; }

stream {
  upstream otlp_grpc {
    server gw1:4317 max_fails=3 fail_timeout=15s;
    server gw2:4317 max_fails=3 fail_timeout=15s;
    server gw3:4317 max_fails=3 fail_timeout=15s;
  }
  server {
    listen 4317;            # gRPC
    proxy_pass otlp_grpc;
    proxy_connect_timeout 1s;
    proxy_timeout 30s;
  }
}

http {
  upstream otlp_http {
    server gw1:4318 max_fails=3 fail_timeout=15s;
    server gw2:4318 max_fails=3 fail_timeout=15s;
    server gw3:4318 max_fails=3 fail_timeout=15s;
  }
  server {
    listen 4318;            # HTTP/JSON
    location / {
      proxy_pass http://otlp_http;
      proxy_next_upstream error timeout http_502 http_503 http_504;
    }
  }
}

Create a ./config directory in the same root directory as your docker-compose.yaml, and create 3 files.

> config/
    config.yaml
    telgen-config.yaml
    logging.yaml

Paste this basic config into the config.yaml and telgen-config.yaml for the BDOT Collector to have a base config to start. I’ll then configure it with Bindplane.

receivers:
  nop:
processors:
  batch:
exporters:
  nop:
service:
  pipelines:
    metrics:
      receivers: [nop]
      processors: [batch]
      exporters: [nop]
  telemetry:
    metrics:
      level: none

And, a base setup for the logging.yaml.

output: stdout
level: info

Start the Docker Compose services.

docker compose up -d

Jump into Bindplane and create three configurations for:

telgen
otlp-lb-gw
external-gw

The telgen configuration has a Telemetry Generator source.

And, an OTLP destination.

The OTLP destination is configured to send telemetry to the otlp-lb hostname, which is the hostname for the Nginx load balancer I’m running in Docker Compose.

Next, the otlp-lb-gw configuration has an OTLP source that listens on 0.0.0.0 and ports 4317 and 4318.

The destination is also OTLP, but instead sending to the external-gw hostname.

Finally, the external-gw configuration is again using an identical OTLP source.

And, a Dev Null destination.

This setup enables you to drop in whatever destination you want in the list of destinations for the external-gw configuration. Go wild! 😂

If you open the processor node for the Dev Null destination, you’ll see logs flowing through the load balancer.

While in the otlp-lb-gw configuration, if you open a processor node, you’ll see evenly distributed load across all three collectors.

That’s how you load balance telemetry across multiple collectors with Nginx.

If you would rather apply these configs via the Bindplane CLI, get the files on GitHub, here.

Load Balancing Exporter

The second option is to use the dedicated loadbalancing exporter in the collector. With this exporter you can specify multiple downstream collectors that will receive the telemetry traffic equally.

One quick note before about the load balancing exporter. You don’t always need it. Its main job is to make sure spans from the same trace stick together and get routed to the same backend collector. That’s super useful for distributed tracing with sampling. But if you’re just shipping logs and metrics, or even traces without fancy sampling rules, you can probably skip it and stick with Nginx.

I’ll set up the architecture just as I did above but with yet another collector instead the Nginx load balancer:

Three Gateway Collectors in parallel
One Gateway Collector using the loadbalancing exporter
One Agent Collector configured to generate telemetry (app simulation)

This behaves identical to an Nginx load balancer. However this requires one less step and less configuration overhead. No need to configure and run Nginx, manage specific Nginx files, instead run one more instance of the collector and use a trusty collector config.yaml that you’re already familiar with.

The drop in replacement for the use case above is as follows. In the docker-compose.yaml replace the otlp-lb Nginx service with another collector named lb.

services:


# ...

  lb:
    image: ghcr.io/observiq/bindplane-agent:1.79.2
    container_name: lb
    hostname: lb
    command: ["--config=/etc/otel/config/lb-config.yaml"]
    volumes:
      - ./config:/etc/otel/config
      - lb-storage:/etc/otel/storage
    ports:
      - "4317:4317"   # OTLP gRPC - external endpoint
      - "4318:4318"   # OTLP HTTP/JSON - external endpoint
    environment:
      OPAMP_ENDPOINT: "wss://app.bindplane.com/v1/opamp"
      OPAMP_SECRET_KEY: "01JFJGVKWHQ1SPQVDGZEHVA995"
      OPAMP_LABELS: ephemeral=true
      MANAGER_YAML_PATH: /etc/otel/config/lb-manager.yaml
      CONFIG_YAML_PATH: /etc/otel/config/lb-config.yaml
      LOGGING_YAML_PATH: /etc/otel/config/logging.yaml
    depends_on: [gw1, gw2, gw3]

# ...

Create a base lb-config.yaml for this collector instance in the ./config directory. Bindplane will update this remotely once you add a destination for the loadbalancing exporter.

receivers:
  nop:
processors:
  batch:
exporters:
  nop:
service:
  pipelines:
    metrics:
      receivers: [nop]
      processors: [batch]
      exporters: [nop]
  telemetry:
    metrics:
      level: none

Go ahead and restart Docker Compose.

docker compose down
docker compose up -d

This will start the new lb collector. In Bindplane, go ahead and create a new configuration called lb and add an OTLP source that listens on 0.0.0.0 and ports 4317 and 4318.

Now, create a custom destination and paste the loadbalancing exporter configuration in the input field.

loadbalancing:
    protocol:
      otlp:
        tls:
          insecure: true
        timeout: 30s
        retry_on_failure:
          enabled: true
          initial_interval: 5s
          max_elapsed_time: 300s
          max_interval: 30s
        sending_queue:
          enabled: true
          num_consumers: 10
          queue_size: 5000
    resolver:
      static:
        hostnames:
          - gw1:4317
          - gw2:4317
          - gw3:4317

Note that the hostnames correlate to the hostnames of the gateway collectors configured in Docker Compose. Save this configuration and roll it out to the new lb collector. Opening the gw configuration in Bindplane and selecting a processor node, you’ll see the telemetry flowing through all 3 gateway collector instances.

You’ll see an even nicer split by seeing the telemetry throughput across all collectors in the Agents view.

The lb and external-gw are reporting the same throughput with the three gateway collectors load balancing traffic equally.

The loadbalancing exporter is behaving like a drop-in replacement for Nginx. I would call that a win. Less configuration overhead, fewer moving parts, and no need to learn specific Nginx configs. Instead, focus only on the collector.

To get this sample up-and-running quickly, apply these configs via the Bindplane CLI, get the files on GitHub, here.

Since you now have a good understanding of how to configure OpenTelemetry Collector infrastructure for high availability, let's move into details about resilience specifically.

Building Resilience into Your Collector

When it comes to resilience, features like retry logic, persistent queues, and batching should be handled in the Agent Collectors. These are the instances sitting closest to your workloads; they’re most at risk of losing data if something goes wrong. The Agent’s job is to collect, buffer, and forward telemetry reliably, even when the backend is flaky or slow.

How you configure the OpenTelemetry collector for resilience to avoid losing telemetry during network issues or telemetry backend outages:

Batching groups signals before export, improving efficiency.
Retry ensures failed exports are re-attempted. For critical workloads, increase max_elapsed_time to tolerate longer outages—but be aware this will increase the buffer size on disk.
Persistent Queue stores retries on disk, protecting against data loss if the Collector crashes. You can configure:
- Number of consumers – how many parallel retry workers run
- Queue size – how many batches are stored
- Persistence – enables disk buffering for reliability

Retry & Persistent Queue

Luckily enough for you, Bindplane handles both retries and the persistent queue out of the box for OTLP exporters.

Take a look at the telgen configuration. This is the collector we’re running in agent-mode simulating a bunch of telemetry traffic.

In the telgen-config.yaml, you'll see OTLP exporter is configured with both the persistent queue and retries.

exporters:
    otlp/lb:
        compression: gzip
        endpoint: gw:4317
        retry_on_failure:
            enabled: true
            initial_interval: 5s
            max_elapsed_time: 300s
            max_interval: 30s
        sending_queue:
            enabled: true
            num_consumers: 10
            queue_size: 5000
            storage: file_storage/lb
        timeout: 30s
        tls:
            insecure: true

This is because the advanced settings for every OTLP exporter in Bindplane have this default configuration enabled.

The persistent queue directory here is the storage directory that we configured by creating a volume in Docker.

# docker-compose.yaml 

...

volumes:
  gw1-storage:      # persistent queue for gateway-1
  gw2-storage:      # persistent queue for gateway-2
  gw3-storage:      # persistent queue for gateway-3
  telgen-storage:   # persistent queue for telemetry generator
  lb-storage:    # persistent queue for load-balancing gateway
  external-gw-storage: # persistent queue for external gateway

...

Bindplane then automatically configures a storage extension in the config and enables it like this:

# telgen-config.yaml 

...
extensions:
    file_storage/lb:
        compaction:
            directory: ${OIQ_OTEL_COLLECTOR_HOME}/storage
            on_rebound: true
        directory: ${OIQ_OTEL_COLLECTOR_HOME}/storage
service:
    extensions:
        - file_storage/lb
...

Note that the OIQ_OTEL_COLLECTOR_HOME environment variable actually is mapped to the /etc/otel directory.

Now your telemetry pipeline becomes resilient and HA-ready with data persistence to survive restarts, persistent queue buffering to handle temporary outages, and failover recovery to prevent data loss.

Batching

Batching is a whole other story, because you need to add a processor on the processor node for it to be enabled before connecting it to the destination.

Agent-mode collectors should batch telemetry before sending it to the gateway collector. The OTLP receiver on the gateway side will receive batches and forward them to your telemetry backend of choice.

In the telgen configuration, click a processor node and add a batch processor.

This config will send a batch of telemetry signals every 200ms regardless of the size. Or, it will send a batch of the size 8192 regardless of the timeout. Applying this processor in Bindplane will generate a config like this:

# telgen-config.yaml 

...

processors:
    batch/lb: null
    batch/lb-0__processor0:
        send_batch_max_size: 0
        send_batch_size: 8192
        timeout: 200ms

...

Kubernetes-native load balancing with HorizontalPodAutoscaler

Finally, after all the breakdowns, explanations, and diagrams, it’s time to show you what it would look like in the wild with a simple Kubernetes sample.

Using Kubernetes is the preferred architecture suggested by the Bindplane team and the OpenTelemetry community. K8s will maximize the benefits you get with Bindplane as well.

I’ll set up the architecture with:

One Agent-mode Collector running per node on the K8s cluster configured to generate telemetry (app simulation)
A Gateway Collector Deployment
- Using a HorizontalPodAutoscaler scaling from 2 to 10 pods
- And a ClusterIP service
- Configured with persistent storage, sending queue, and retry
An external Gateway Collector running on another cluster acting as a mock telemetry backend

Luckily enough getting all the K8s YAML manifests for the collectors is all point-and-click from the Bindplane UI. However, you need to build the configurations first, before applying the collectors to your K8s cluster.

For the sake of simplicity l’ll show how to spin up two K8s clusters with kind, and use them in this demo.

kind create cluster --name kind-2
kind create cluster --name kind-1
# make sure you set the context to the kind-1 cluster first
kubectl config use-context kind-kind-1

Next, jump into Bindplane and create three configurations for:

telgen-kind-1
gw-kind-1
external-gw-kind-2

The telgen-kind-1 configuration has a Custom source with a telemetrygeneratorreceiver.

telemetrygeneratorreceiver:
        generators:
            - additional_config:
                body: 127.0.0.1 - - [30/Jun/2025:12:00:00 +0000] \"GET /index.html HTTP/1.1\" 200 512
                severity: 9
              type: logs
        payloads_per_second: 1

And, a Bindplane Gateway destination.

Note: This is identical to any OTLP destination.

The Bindplane Gateway destination is configured to send telemetry to the bindplane-gateway-agent.bindplane-agent.svc.cluster.local

hostname, which is the hostname for the Bindplane Gateway Collector service in Kubernetes that you’ll start in a second.

The final step for this configuration is to click a processor node and add a batch processor.

Next, the gw-kind-1 configuration has a Bindplane Gateway source that listens on 0.0.0.0 and ports 4317 and 4318.

The destination is OTLP, and sending telemetry to the IP address (172.18.0.2) of the external gateway running on the second K8s cluster.

Note: This might differ for your clusters. If you are using kind, like I am in this demo, the IP will be 172.18.0.2.

Finally, the external-gw-kind-2 configuration is again using an OTLP source.

And, a Dev Null destination.

Feel free to use the Bindplane CLI and these resources to apply all the configurations in one go without having to do it manually in the UI.

With the configurations created, you can install collectors easily by getting manifest files from your Bindplane account. Navigate to the install agents UI in Bindplane and select a Kubernetes environment. Use the Node platform and telgen-kind-1 configuration.

Clicking next will show a manifest file for you to apply in the cluster.

Save this file as node-agent-kind-1.yaml. Check out below what a sample of it looks like. Or, see the file in GitHub, here.

---
apiVersion: v1
kind: Namespace
metadata:
  labels:
    app.kubernetes.io/name: bindplane-agent
  name: bindplane-agent
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/name: bindplane-agent
  name: bindplane-agent
  namespace: bindplane-agent
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: bindplane-agent
  labels:
    app.kubernetes.io/name: bindplane-agent
rules:
- apiGroups:
  - ""
  resources:
  - events
  - namespaces
  - namespaces/status
  - nodes
  - nodes/spec
  - nodes/stats
  - nodes/proxy
  - pods
  - pods/status
  - replicationcontrollers
  - replicationcontrollers/status
  - resourcequotas
  - services
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - daemonsets
  - deployments
  - replicasets
  - statefulsets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - daemonsets
  - deployments
  - replicasets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - batch
  resources:
  - jobs
  - cronjobs
  verbs:
  - get
  - list
  - watch
- apiGroups:
    - autoscaling
  resources:
    - horizontalpodautoscalers
  verbs:
    - get
    - list
    - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: bindplane-agent
  labels:
    app.kubernetes.io/name: bindplane-agent
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: bindplane-agent
subjects:
- kind: ServiceAccount
  name: bindplane-agent
  namespace: bindplane-agent
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: bindplane-agent
  name: bindplane-node-agent
  namespace: bindplane-agent
spec:
  ports:
  - appProtocol: grpc
    name: otlp-grpc
    port: 4317
    protocol: TCP
    targetPort: 4317
  - appProtocol: http
    name: otlp-http
    port: 4318
    protocol: TCP
    targetPort: 4318
  selector:
    app.kubernetes.io/name: bindplane-agent
    app.kubernetes.io/component: node
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: bindplane-agent
    app.kubernetes.io/component: node
  name: bindplane-node-agent-headless
  namespace: bindplane-agent
spec:
  clusterIP: None
  ports:
  - appProtocol: grpc
    name: otlp-grpc
    port: 4317
    protocol: TCP
    targetPort: 4317
  - appProtocol: http
    name: otlp-http
    port: 4318
    protocol: TCP
    targetPort: 4318
  selector:
    app.kubernetes.io/name: bindplane-agent
    app.kubernetes.io/component: node
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: bindplane-node-agent-setup
  labels:
    app.kubernetes.io/name: bindplane-agent
    app.kubernetes.io/component: node
  namespace: bindplane-agent
data:
  # This script assumes it is running in /etc/otel.
  setup.sh: |
    # Configure storage/ emptyDir volume permissions so the
    # manager configuration can ge written to it.
    chown 10005:10005 storage/

    # Copy config and logging configuration files to storage/
    # hostPath volume if they do not already exist.
    if [ ! -f storage/config.yaml ]; then
      echo '
      receivers:
        nop:
      processors:
        batch:
      exporters:
        nop:
      service:
        pipelines:
          metrics:
            receivers: [nop]
            processors: [batch]
            exporters: [nop]
        telemetry:
          metrics:
            level: none
      ' > storage/config.yaml
    fi
    if [ ! -f storage/logging.yaml ]; then
      echo '
      output: stdout
      level: info
      ' > storage/logging.yaml
    fi
    chown 10005:10005 storage/config.yaml storage/logging.yaml
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: bindplane-node-agent
  labels:
    app.kubernetes.io/name: bindplane-agent
    app.kubernetes.io/component: node
  namespace: bindplane-agent
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: bindplane-agent
      app.kubernetes.io/component: node
  template:
    metadata:
      labels:
        app.kubernetes.io/name: bindplane-agent
        app.kubernetes.io/component: node
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: /metrics
        prometheus.io/port: "8888"
        prometheus.io/scheme: http
        prometheus.io/job-name: bindplane-node-agent
    spec:
      serviceAccount: bindplane-agent
      initContainers:
        - name: setup
          image: busybox:latest
          securityContext:
            # Required for changing permissions from
            # root to otel user in emptyDir volume.
            runAsUser: 0
          command: ["sh", "/setup/setup.sh"]
          volumeMounts:
            - mountPath: /etc/otel/config
              name: config
            - mountPath: /storage
              name: storage
            - mountPath: "/setup"
              name: setup
      containers:
        - name: opentelemetry-collector
          image: ghcr.io/observiq/bindplane-agent:1.80.1
          imagePullPolicy: IfNotPresent
          securityContext:
            readOnlyRootFilesystem: true
            # Required for reading container logs hostPath.
            runAsUser: 0
          ports:
            - containerPort: 8888
              name: prometheus
          resources:
            requests:
              memory: 200Mi
              cpu: 100m
            limits:
              memory: 200Mi
          env:
            - name: OPAMP_ENDPOINT
              value: wss://app.bindplane.com/v1/opamp
            - name: OPAMP_SECRET_KEY
              value: <secret>
            - name: OPAMP_AGENT_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: OPAMP_LABELS
              value: configuration=telgen-kind-1,container-platform=kubernetes-daemonset,install_id=0979c5c2-bd7a-41c1-89b8-2c16441886ab
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            # The collector process updates config.yaml
            # and manager.yaml when receiving changes
            # from the OpAMP server.
            #
            # The config.yaml is persisted by saving it to the
            # hostPath volume, allowing the agent to continue
            # running after restart during an OpAMP server outage.
            #
            # The manager configuration must be re-generated on
            # every startup due to how the bindplane-agent handles
            # manager configuration. It prefers a manager config file
            # over environment variables, meaning it cannot be
            # updated using environment variables, if it is persisted).
            - name: CONFIG_YAML_PATH
              value: /etc/otel/storage/config.yaml
            - name: MANAGER_YAML_PATH
              value: /etc/otel/config/manager.yaml
            - name: LOGGING_YAML_PATH
              value: /etc/otel/storage/logging.yaml
          volumeMounts:
            - mountPath: /etc/otel/config
              name: config
            - mountPath: /run/log/journal
              name: runlog
              readOnly: true
            - mountPath: /var/log
              name: varlog
              readOnly: true
            - mountPath: /var/lib/docker/containers
              name: dockerlogs
              readOnly: true
            - mountPath: /etc/otel/storage
              name: storage
      volumes:
        - name: config
          emptyDir: {}
        - name: runlog
          hostPath:
            path: /run/log/journal
        - name: varlog
          hostPath:
            path: /var/log
        - name: dockerlogs
          hostPath:
            path: /var/lib/docker/containers
        - name: storage
          hostPath:
            path: /var/lib/observiq/otelcol/container
        - name: setup
          configMap:
            name: bindplane-node-agent-setup

In short, this manifest deploys the BDOT Collector as a DaemonSet on every node, using OpAMP to receive config from Bindplane. It includes:

RBAC to read Kubernetes objects (pods, nodes, deployments, etc.)
Services to expose OTLP ports (4317 gRPC, 4318 HTTP)
An init container to bootstrap a config to start the collector which will be replaced by the telgen-kind-1 configuration once started
Persistent hostPath storage for retries and disk buffering
Prometheus annotations for metrics scraping

Your file will include the correct OPAMP_ENDPOINT, OPAMP_SECRET_KEY, and OPAMP_LABELS.

Go ahead and apply this manifest to the first k8s cluster.

kubectl config use-context kind-kind-1
kubectl apply -f node-agent-kind-1.yaml

Now, install another collector in the K8s cluster, but now choose a Gateway and the gw-kind-1 configuration.

You’ll get a manifest file to apply again, but this time a deployment. Save it as gateway-collector-kind-1.yaml. Here’s what it looks like in GitHub.

Here’s the full manifest as a deployment with a horizontal pod autoscaler.

---
apiVersion: v1
kind: Namespace
metadata:
  labels:
    app.kubernetes.io/name: bindplane-agent
  name: bindplane-agent
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/name: bindplane-agent
  name: bindplane-agent
  namespace: bindplane-agent
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: bindplane-agent
    app.kubernetes.io/component: gateway
  name: bindplane-gateway-agent
  namespace: bindplane-agent
spec:
  ports:
  - appProtocol: grpc
    name: otlp-grpc
    port: 4317
    protocol: TCP
    targetPort: 4317
  - appProtocol: http
    name: otlp-http
    port: 4318
    protocol: TCP
    targetPort: 4318
  - appProtocol: tcp
    name: splunk-tcp
    port: 9997
    protocol: TCP
    targetPort: 9997
  - appProtocol: tcp
    name: splunk-hec
    port: 8088
    protocol: TCP
    targetPort: 8088
  selector:
    app.kubernetes.io/name: bindplane-agent
    app.kubernetes.io/component: gateway
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: bindplane-agent
    app.kubernetes.io/component: gateway
  name: bindplane-gateway-agent-headless
  namespace: bindplane-agent
spec:
  clusterIP: None
  ports:
  - appProtocol: grpc
    name: otlp-grpc
    port: 4317
    protocol: TCP
    targetPort: 4317
  - appProtocol: http
    name: otlp-http
    port: 4318
    protocol: TCP
    targetPort: 4318
  selector:
    app.kubernetes.io/name: bindplane-agent
    app.kubernetes.io/component: gateway
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bindplane-gateway-agent
  labels:
    app.kubernetes.io/name: bindplane-agent
    app.kubernetes.io/component: gateway
  namespace: bindplane-agent
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: bindplane-agent
      app.kubernetes.io/component: gateway
  template:
    metadata:
      labels:
        app.kubernetes.io/name: bindplane-agent
        app.kubernetes.io/component: gateway
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: /metrics
        prometheus.io/port: "8888"
        prometheus.io/scheme: http
        prometheus.io/job-name: bindplane-gateway-agent
    spec:
      serviceAccount: bindplane-agent
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                topologyKey: kubernetes.io/hostname
                labelSelector:
                  matchExpressions:
                    - key: app.kubernetes.io/name
                      operator: In
                      values:  [bindplane-agent]
                    - key: app.kubernetes.io/component
                      operator: In
                      values: [gateway]
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000000000
        runAsGroup: 1000000000
        fsGroup: 1000000000
        seccompProfile:
          type: RuntimeDefault
      initContainers:
        - name: setup-volumes
          image: ghcr.io/observiq/bindplane-agent:1.80.1
          securityContext:
            runAsNonRoot: true
            runAsUser: 1000000000
            runAsGroup: 1000000000
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            seccompProfile:
              type: RuntimeDefault
            capabilities:
              drop:
                - ALL
          command:
            - 'sh'
            - '-c'
            - |
              echo '
              receivers:
                nop:
              processors:
                batch:
              exporters:
                nop:
              service:
                pipelines:
                  metrics:
                    receivers: [nop]
                    processors: [batch]
                    exporters: [nop]
                telemetry:
                  metrics:
                    level: none
              ' > /etc/otel/storage/config.yaml
              echo '
              output: stdout
              level: info
              ' > /etc/otel/storage/logging.yaml
          resources:
            requests:
              memory: 200Mi
              cpu: 100m
            limits:
              memory: 200Mi
          volumeMounts:
            - mountPath: /etc/otel/storage
              name: bindplane-gateway-agent-storage
      containers:
        - name: opentelemetry-container
          image: ghcr.io/observiq/bindplane-agent:1.80.1
          imagePullPolicy: IfNotPresent
          securityContext:
            runAsNonRoot: true
            runAsUser: 1000000000
            runAsGroup: 1000000000
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            seccompProfile:
              type: RuntimeDefault
            capabilities:
              drop:
                - ALL
          resources:
            requests:
              memory: 500Mi
              cpu: 250m
            limits:
              memory: 500Mi
          ports:
            - containerPort: 8888
              name: prometheus
          env:
            - name: OPAMP_ENDPOINT
              value: wss://app.bindplane.com/v1/opamp
            - name: OPAMP_SECRET_KEY
              value: <secret>
            - name: OPAMP_AGENT_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: OPAMP_LABELS
              value: configuration=gw-kind-1,container-platform=kubernetes-gateway,install_id=51dbe4d2-83d2-45c0-ab4a-e0c127a59649
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            # The collector process updates config.yaml
            # and manager.yaml when receiving changes
            # from the OpAMP server.
            - name: CONFIG_YAML_PATH
              value: /etc/otel/storage/config.yaml
            - name: MANAGER_YAML_PATH
              value: /etc/otel/config/manager.yaml
            - name: LOGGING_YAML_PATH
              value: /etc/otel/storage/logging.yaml
          volumeMounts:
          - mountPath: /etc/otel/storage
            name: bindplane-gateway-agent-storage
          - mountPath: /etc/otel/config
            name: config
      volumes:
        - name: config
          emptyDir: {}
        - name: bindplane-gateway-agent-storage
          emptyDir: {}
      # Allow exporters to drain their queue for up to
      # five minutes.
      terminationGracePeriodSeconds: 500
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: bindplane-gateway-agent
  namespace: bindplane-agent
spec:
  maxReplicas: 10
  minReplicas: 2
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: bindplane-gateway-agent
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60

Here’s a breakdown of what this manifest does:

Creates a dedicated namespace and service account for the Bindplane Gateway Collector (bindplane-agent).
Defines two Kubernetes services:
- A standard ClusterIP service for OTLP (gRPC/HTTP) and Splunk (TCP/HEC) traffic.
- A headless service for direct pod discovery, useful in peer-to-peer setups.
Deploys the Bindplane Agent as a scalable Deployment:
- Runs the OpenTelemetry Collector image.
- Bootstraps basic config via an initContainer.
- Secure runtime with strict securityContext settings.
- Prometheus annotations enable metrics scraping.
Auto-scales the collector horizontally using an HPA:
- Scales between 2 and 10 replicas based on CPU utilization.
Uses OpAMP to receive remote config and updates from Bindplane.
Mounts ephemeral storage for config and persistent queue support using emptyDir.

Your file will include the correct OPAMP_ENDPOINT, OPAMP_SECRET_KEY, and OPAMP_LABELS.

Apply it in the first k8s cluster.

kubectl apply -f gateway-collector-kind-1.yaml

Now, create an identical Gateway Collector as above but use the external-gw-kind-2 configuration.

You’ll get a manifest file to apply again, but this time apply it in your second cluster. Save it as gateway-collector-kind-2.yaml. Here’s what it looks like in GitHub. I won’t bother showing you the manifest YAML since it will be identical to the one above.

kubectl config use-context kind-kind-2
kubectl apply -f gateway-collector-kind-2.yaml

Finally, to expose this Gateway Collector’s service and enable OTLP traffic from cluster 1 to cluster 2 I’ll use this NodePort service called gateway-nodeport-service.yaml.

apiVersion: v1
kind: Service
metadata:
  name: bindplane-gateway-agent-nodeport
  namespace: bindplane-agent
  labels:
    app.kubernetes.io/name: bindplane-agent
    app.kubernetes.io/component: gateway
spec:
  type: NodePort
  ports:
  - name: otlp-grpc
    port: 4317
    targetPort: 4317
    nodePort: 30317
    protocol: TCP
  - name: otlp-http
    port: 4318
    targetPort: 4318
    nodePort: 30318
    protocol: TCP
  selector:
    app.kubernetes.io/name: bindplane-agent
    app.kubernetes.io/component: gateway

And, apply it with:

kubectl apply -f gateway-nodeport-service.yaml

Your final setup will look like this.

One Agent-mode collector sending telemetry traffic via a horizontally scaled Gateway-mode collector to an external Gateway running in a separate cluster. This can be any other telemetry backend of your choice.

You’ll have 5 collectors running in total.

And, 3 configurations, where two of them will be scaled between 2 and 10 collector pods.

To get this sample up-and-running quickly, apply these configs via the Bindplane CLI, get the files on GitHub, here.

Final thoughts

At the end of the day, high availability for the OpenTelemetry Collector means one thing: don’t lose telemetry when stuff breaks.

You want things to keep working when a telemetry backend goes down, a node restarts, or you’re pushing out updates. That’s why the Agent-Gateway pattern exists. That’s why we scale horizontally. That’s why we use batching, retries, and persistent queues.

Where?

We’re putting a ribbon on the Launch Week with a final blog post, video demo, and live stream.

How to Join

Inbox: A blog post announcement with a demo video will hit your email inbox every morning at 10:30 a.m. ET. Sign up for the newsletter here.
Social: Follow @Bindplane on LinkedIn & X for teasers and to join the live streams.
Live: Live streams will start at 11 a.m. ET each day. Jump into the daily stream, ask questions, and watch feature demos in real time.

Clear your calendar, charge the coffee mug, and get ready for a week with five new feature releases. Launch Week starts Monday, June 2nd—see you there.

End-to-End Observability with Grafana LGTM Stack

Adnan Rahić — Fri, 11 Oct 2024 14:37:47 +0000

Ensuring your applications are running smoothly requires more than just monitoring. It demands comprehensive visibility into every aspect of your system, from logs and metrics to traces. This is where end-to-end observability comes into play. It's about connecting the dots between different parts of your system to get a complete picture of how everything is performing.

In this blog post, we'll dive into setting up end-to-end observability using Grafana and its associated tools, including OpenTelemetry, Prometheus, Loki, and Tempo. Let's break down each component and see how they fit together to provide a robust observability solution.

View the full sample code for the observability stack you'll build, here.

What is End-to-End Observability, And Why Do We Need It?

Before we dive into the setup, it's important to understand what end-to-end observability means and why it's essential. End-to-end observability provides a unified view of your entire system, including traces, metrics, and logs. It helps you monitor and debug your applications more effectively by correlating different types of data to understand the overall health of your system.

With Grafana playing a key role in visualizing and analyzing this data, we can bring everything together. Grafana's ability to integrate with multiple data sources allows us to correlate logs, metrics, and traces in one unified platform, making troubleshooting and maintaining your system easier. The architecture diagram below gives a clear view of how these components fit together to provide end-to-end observability.

Here's what is happening:

The application is instrumented using OpenTelemetry for metrics and traces and Winston-Loki for logging.
The application sends logs to Loki, metrics to Prometheus, and trace data to Tempo.
Tempo forwards traces to the Tracetest Agent.
Grafana is used to visualize the logs (from Loki), metrics (from Prometheus), and traces (from Tempo), providing a unified observability dashboard.
Finally, the Tracetest Agent syncs the data to the Tracetest UI for trace-based testing.

But how do we get all this data into Grafana? The first step is to instrument your application. This means directly embedding tracing, logging, and metrics functionality into your code. By doing this, you'll generate the metrics required to monitor the system and identify any performance bottlenecks or issues.

1. Setting Up Instrumentation in Your Application

The first step in setting up end-to-end observability is to instrument your application. This involves adding tracing, logging, and metrics SDKs or libraries to your application code.

For this tutorial, you'll set up a simple Express server in your root directory in index.js.

const express = require('express')
const app = express()
const port = 8081

app.get('/', (req, res) => {
  res.send('Hello World!')
})

app.listen(port, () => {
 console.log(`Example app listening on port ${port}`)
})

This is a basic web application; while it doesn't include observability yet, let's imagine this is your real-world app or service that you've built.

1. Adding Metrics to Your Application

Metrics gives you insights into the behavior and performance of your application by tracking things like response times, error rates, or even custom-defined metrics like the number of requests handled.

In observability, metrics help identify trends or issues in real-time, allowing you to react quickly to problems. For this, we'll use OpenTelemetry, a popular open-source observability framework, to instrument our application for collecting metrics.

To start collecting metrics, create a new file, meter.js, which will handle generating and exporting the metrics to Prometheus, a time-series database commonly used for metrics storage and querying.

// meter.js
const { MeterProvider } = require('@opentelemetry/sdk-metrics');
const { PrometheusExporter } = require('@opentelemetry/exporter-prometheus');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');

const prometheusExporter = new PrometheusExporter({
port: 9464,
endpoint: '/metrics',
}, () => {
 console.log('Prometheus scrape endpoint: http://localhost:9464/metrics');
});

const meterProvider = new MeterProvider({
resource: new Resource({
[SemanticResourceAttributes.SERVICE_NAME]: 'hello-world-app',
  }),
});

meterProvider.addMetricReader(prometheusExporter);
const meter = meterProvider.getMeter('hello-world-meter');

module.exports = meter;

By adding this meter.js file to our application, we're setting the groundwork to collect and export metrics. Prometheus will now be able to scrape these metrics and store them for visualization and alerting later in Grafana.

2. Adding Logging to Your Application

Logs are a key component of observability, providing real-time information about the events happening inside your application. They help you trace errors, track user activity, and understand the execution flow. By centralizing your logs, you can more easily search, filter, and correlate them with other observability data like metrics and traces.

To manage logs, you'll use Winston, a popular logging library for Node.js, along with the Winston-Loki transport, which sends logs directly to a Loki server. Loki is a log aggregation system designed to work alongside tools like Prometheus, providing a scalable way to collect and query logs.

Create a logger.js file to handle logging and send the logs to a Loki server.

// logger.js
const winston = require('winston');
const LokiTransport = require('winston-loki');

const logger = winston.createLogger({
level: 'info',
  format: winston.format.json(),
  transports: [
 new LokiTransport({
host: 'http://loki:3100', // Loki URL
labels: { job: 'loki-service' },
json: true,
batching: true,
interval: 5,
    }),
  ],
});

module.exports = logger;

Now, import the meter and logger in index.js to add logs and metrics to the application.

const express = require('express');
const logger = require('./logger');
const meter = require('./meter');

const app = express();

// Define a custom metric (e.g., a request counter)
const requestCounter = meter.createCounter('http_requests', {
description: 'Counts HTTP requests',
});

// Middleware to increment the counter on every request and log the visited URL
app.use((req, res, next) => {
  logger.info(`Received request for ${req.url}`);
  requestCounter.add(1, { method: req.method, route: req.path });
  next();
});

app.get('/', (req, res) => {
 // Simulate some work
  setTimeout(() => {
    res.send('Hello, World!');
}, 100);
});

// Start the server
app.listen(8081, () => {
  logger.info('Server is running on http://localhost:8081');
});

3. Adding Traces to Your Application

Traces are a critical part of understanding how requests flow through your application. They give you the ability to track requests from the moment they enter the system to when they leave, allowing you to see:

The services that were called.
The time it took for each of the services.
The places where bottlenecks may be occurring.

This is especially important for debugging performance issues and understanding distributed systems.

To create and collect these traces, we'll use the OpenTelemetry SDK along with auto-instrumentation, which automatically instruments Node.js modules to start generating traces without you having to manually add trace code everywhere.

Create a file called tracer.js to set up OpenTelemetry in your application.

// tracer.js
const opentelemetry = require('@opentelemetry/sdk-node')
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node')
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http')
const { ConsoleSpanExporter } = require('@opentelemetry/sdk-trace-node');
const dotenv = require("dotenv")
dotenv.config()

const sdk = new opentelemetry.NodeSDK({
traceExporter: new OTLPTraceExporter({ url: process.env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
}),
  instrumentations: [getNodeAutoInstrumentations()],
})
sdk.start()

And a .env file to load the OTEL_EXPORTER_OTLP_TRACES_ENDPOINT environment variable.

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://tempo:4318/v1/traces"

Once this code is in place, and tracer.js is preloaded when running your index.js, every request to your Node.js app will automatically generate traces, thanks to OpenTelemetry's auto-instrumentation. These traces are collected and sent to an OTLP-compatible backend, which is Tempo in this case.

4. Containerize the Application

To run your application in a container, create a Dockerfile in your root directory:

FROM node:slim
WORKDIR /usr/src/app/
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE  8081

Build the image of your application with the command docker build -t <dockerhub-username>/tracetest-app . in the root directory and push it to DockerHub with docker push <dockerhub-username>/tracetest-app
Also, add a new script in your package.json to automatically require tracer.js when running the server.

"scripts": {
  "index-with-tracer": "node -r ./tracer.js index.js"
},

Now create a docker-compose.yml and add the the code below to run the application in the container.

services:
  app:
    image: <your-dockerhub-uername>/tracetest-app
    build: .
    command: npm run index-with-tracer
    ports:
      -  "8081:8081"
    environment:
      -  OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=${OTEL_EXPORTER_OTLP_TRACES_ENDPOINT}
    depends_on:
      tempo:
        condition: service_started
      tracetest-agent:
        condition: service_started

And with that, you have successfully instrumented and containerized your application. Let's focus on setting up the Prometheus, Loki, and Tempo servers to collect all the data and visualize it in Grafana.

2. Setting up Prometheus for the Node.js Application

Prometheus is an open-source monitoring and alerting toolkit designed for reliability and scalability. It collects metrics from various sources, storing them in a time-series database, and providing a powerful query language (PromQL) for analysis.

In the context of observability, Prometheus is used to gather metrics from your applications and infrastructure, which can then be visualized and analyzed to gain insights into system performance and health.

To integrate Prometheus with your Node.js application, follow these steps:

1. Create the Prometheus Configuration File

The prometheus.yml file specifies how Prometheus should discover and scrape metrics from your application. Here's the configuration:

# prometheus.yml
global:
  scrape_interval: 5s

scrape_configs:
  - job_name: 'hello-world-app'
    static_configs:
      - targets: ['app:9464'] # Metrics exposed on port 9464

This defines the target endpoints where Prometheus will look for metrics. host.docker.internal refers to your local Docker host, and 9464 is the port on which your Node.js application exposes the metrics.

2. Run Prometheus in a Docker Container

To run the Prometheus server in a container, add the below service in your docker-compose.yml.

prometheus:
  image: prom/prometheus
  volumes:
    -  ./prometheus.yml:/etc/prometheus/prometheus.yml
  ports:
    -  "9090:9090"

Here, image specifies the Docker image to use for Prometheus and volumes mounts your local prometheus.yml file into the container at etc/prometheus/prometheus.yml. This file contains the configuration Prometheus will use and ports maps port 9090 on the Docker container to port 9090 on your host machine. This makes the Prometheus web UI accessible at localhost:9090.

Use Docker Compose to start the Prometheus server: docker-compose up

3. Verifying Metrics Collection

Open your browser and navigate to http://localhost:9090/graph to access the Prometheus web UI. Here, you can execute queries to verify that Prometheus is successfully scraping and storing metrics.

For instance, running the query http_requests_total will display metrics related to the number of HTTP requests processed by your application.

With Prometheus configured, you now have a monitoring tool that collects and stores metrics from your Node.js application. Now, let's configure Loki to get logs from your application.

3. Setting up Loki for the Node.js Application

Logs are vital for diagnosing issues and understanding how your application behaves in different scenarios. They offer detailed insights into system events, errors, and application flow, making them indispensable for effective troubleshooting and performance monitoring. Loki is a log aggregation system designed to work seamlessly with Grafana. It collects and stores logs, enabling powerful querying and visualization through Grafana.

To set up Loki to collect logs from your Node.js application, follow these steps:

1. Add Loki to Your Docker Compose Configuration

Extend your docker-compose.yml file to include Loki. This allows you to run Loki as a container alongside your Prometheus instance:

loki:
    image: grafana/loki:2.9.10
    ports:
      -  "3100:3100"

grafana/loki:2.9.10 is the official Loki image from Grafana and the port 3100 on the Docker container is mapped to the port 3100 on your host machine. This allows you to access Loki's API and check its status.

With Loki added to your Docker Compose configuration, restart your containers to include Loki.

docker-compose up

2. Verify Loki's is Running

Loki does not have a traditional web UI for interacting with logs, so you'll need to check its status and ensure it's running correctly by accessing its metrics endpoint.

http://localhost:3100/metrics

This endpoint will display raw metrics data related to Loki, confirming that Loki is up and running. The metrics here are used internally by Grafana to visualize logs and track Loki's performance.

With logs being collected and stored by Loki, you can use Grafana to visualize these logs, helping you gain deeper insights into your application's behavior and troubleshoot issues more effectively.

4. Setting up Tempo for the Node.js Application

While metrics tell you how your application is performing and logs show what is happening, traces help answer why something is going wrong by showing the detailed flow of requests through your system. Tempo is designed to collect and store these traces.

To ensure traces from your Node.js application are captured, configure Tempo to accept traces exported by the OpenTelemetry. Tempo will act as a bridge, collecting traces from your app, which will be fetched by Grafana and the Tracetest Agent for further testing.

1. Create a Tempo Configuration File

Ensure that Tempo is set up with the appropriate storage configuration in the tempo.yaml file to handle the incoming traces.

stream_over_http_enabled: true
server:
  http_listen_port: 80
  log_level: info

query_frontend:
  search:
    duration_slo: 5s
    throughput_bytes_slo: 1.073741824e+09
  trace_by_id:
    duration_slo: 5s

distributor:
  receivers:
    jaeger:
      protocols:
        thrift_http:
        grpc:
        thrift_binary:
        thrift_compact:
    zipkin:
    otlp:
      protocols:
        http:
        grpc:
    opencensus:

ingester:
  max_block_duration: 5m 

compactor:
  compaction:
    block_retention: 1h

storage:
  trace:
    backend: local
    wal:
      path: /var/tempo/wal
    local:
      path: /var/tempo/blocks

2. Add Tempo to Your Docker Compose

In the docker-compose.yml, add Tempo in the services and map the local port 3200 of your machine with 80 of the Docker container since that is the default port of Tempo.

init:
  image: &tempoImage  grafana/tempo:latest
  user: root
  entrypoint:
    -  "chown"
    -  "10001:10001"
    -  "/var/tempo"
  volumes:
    -  ./tempo-data:/var/tempo

tempo:
  image: *tempoImage
  command: [  "-config.file=/etc/tempo.yaml"  ]
  volumes:
    -  ./tempo.yaml:/etc/tempo.yaml
    -  ./tempo-data:/var/tempo
  ports:
    -  "14268:14268"
    -  "3200:80"  # tempo
    -  "9095:9095"
    -  "4417:4317" # otlp grpc
    -  "4418:4318" # otlp http
    -  "9411:9411"
  depends_on:
    -  init

The init container runs a command to change the ownership of the /var/tempo directory to user 10001, and the Tempo service uses the grafana/tempo image to start Tempo with the specified configuration file, exposing multiple ports for tracing protocols.

5. Enabling Observability in Grafana

With Prometheus, Loki, and Tempo set up, it's time to integrate them into Grafana for visualization.

Begin with adding Prometheus as a data source in Grafana. Go to localhost:3000 where Grafana is running. On the login page, enter admin in both username and password.

On the next page, you can update your password or skip it to continue with the default one.

1. Adding a Data Source in Grafana

After logging in, go to the Data Sources, click "Add new Data Source" and search for Prometheus to configure it.

In the "Connection" section enter the Prometheus server URL as http://prometheus:9090. Keep the other settings as default.

Scroll down and click the "Save & test" button to verify the connection between Prometheus and Grafana.

Similarly, search for Loki in the new Data Sources to configure it. In the connection URL of Loki, enter http://loki:3100 and verify it by clicking "Save & test".

Add Tempo as the final Data Source. Configure its Connection URL as http://tempo:80. Finally, verify the server connection by clicking on "Save & test".

Adding these integrations allows you to create dashboards that visualize metrics, logs, and traces all in one place.

2. Setting up Grafana for Visualization

Go to the Dashboards tab to create dashboards in Grafana to visualize your metrics, logs, and traces. For example, you might create a dashboard that shows HTTP request rates, info logs, and trace latencies.

On the next panel, click on "Add visualization" to add panels of Prometheus, Loki and Tempo.

From the available data sources, select Prometheus to configure its panel. In the query section, select http_request_total and run the query to get a time series graph of total HTTP requests on the application. Click the "Save" button to save its configuration and get the panel on the dashboard.

On the dashboard, you will see the panel showing the total HTTP requests on the application in the form of a time series graph. Now, select "Visualization" on the "Add" dropdown to add Loki in the similar way.

In the Data Source, select Loki, select job in the filter, and table as the visualization in the top right corner. Finally, apply the changes to save the panel.

Similar to Loki, add one more panel, select the Data Source as Tempo and visualization as Table to get a proper view of all the traces generated in the application.

With all the three panels created, you can resize and adjust their position on the Grafana dashboard according to your requirements.

6. End-to-End Testing Using Trace-Based Testing with Tracetest

Once you have your observability stack up and running, the next step is ensuring everything works as expected. But how can you test if all the traces, logs, and metrics you're collecting are not only being captured correctly but also providing meaningful insight? That's where Tracetest comes in. It's designed to take your end-to-end testing to the next level by leveraging trace-based testing.

1. Sign up to Tracetest

Let us start by signing up to Tracetest. Go to the Tracetest Sign Up page and log in with your Google or GitHub account.

Create a new organization in your Tracetest account.

In the organization, a new environment must be created as well.

2. Set up the Tracetest Agent

Return to your docker-compose.yml file, and add a service to run the Tracetest Agent in a container.

tracetest-agent:
    image: kubeshop/tracetest-agent
    environment:
      - TRACETEST_API_KEY=${TRACETEST_TOKEN}
      - TRACETEST_ENVIRONMENT_ID=${TRACETEST_ENVIRONMENT_ID}

Find your TRACETEST_TOKEN and TRACETEST_ENVIRONMENT_ID, here.

Now, update the .env file in the root directory and add the values of TRACETEST_API_KEY and TRACETEST_ENVIRONMENT_ID you copied from https://app.tracetest.io/retrieve-token

TRACETEST_TOKEN="<your-tracetest-organization-token>"
TRACETEST_ENVIRONMENT_ID="<your-environment-id>"

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://tempo:4318/v1/traces"

Run the Docker Compose file again with docker-compose up to run the Tracetest Agent with the rest of the services.

3 . Ingest the Traces from Tempo to Tracetest

In the Settings of Tracetest UI, go to the "Trace Ingestion" tab and select Tempo as your tracing backend. Enable trace ingestion, Select connection type as Http, and enter http://tempo:80 as the URL. Finally, click "Test Connection" and "Save" to test the connection with Tempo and save the backend configuration.

You can also apply it with the CLI. First, configure the Tracetest CLI in your Terminal.

tracetest configure --token <your-tracetest-organization-token> --environment <your-environment-id>

Create a file for the connection to Tempo.

# tracetest-trace-ingestion.yaml
type: DataStore
spec:
  id: current
  name: Grafana Tempo
  type: tempo
  default: true
  tempo:
    type: http
    http:
      url: http://tempo:80
      tls:
        insecure: true

And run this command to apply it.

tracetest apply datastore -f tracetest-trace-ingestion.yaml

Now trigger a test with a GET request on http://app:8081 where the application is running.

Go to the Trace tab to get a flow of your traces in the application.

4. Automate the Tests with Tracetest CLI

Now, let us see how to automate these tests. Go to the Automate tab and follow the CLI configuration steps to automate the testing using your command line.

You can download the untitled.yaml, rename it to tracetest-test.yaml, and add assertions to the file in yaml format to execute the tracetest run test --file tracetest-test.yaml --output pretty command and automate the tests.

For example, you can add assertions in the configuration file as given below.

type: Test
spec:
  id: eKmofseIR
  name: Untitled
  trigger:
    type: http
    httpRequest:
      method: GET
      Url: http://app:8081
      headers:
      - key: Content-Type
        value: application/json
  selector: span[tracetest.span.type  =  "http"]
      # the assertions define the checks to be run. In this case, all
      # http spans will be checked for a status code = 200
      - assertions:
          - http.status_code  =  200

Trigger the test.

tracetest run test --file ./tracetest-test.yaml --output pretty

✔ RunGroup: #v52GDwRHR (https://app.tracetest.io/organizations/_HDptBgNg/environments/ttenv_6da9b4f817b8b9df/run/v52GDwRHR)\
Summary: 1 passed, 0 failed, 0 pending\
  ✔ Untitled (https://app.tracetest.io/organizations/_HDptBgNg/environments/ttenv_6da9b4f817b8b9df/test/mMtWIwgNg/run/5/test) - trace id: 4362a098956f9bf8790fc4b37e1ad99f

That's how Tracetest helps you with your End-to-End Observability by testing all generated traces and automating the end-to-end testing pipeline by leveraging telemetry data.

Conclusion

End-to-end observability is critical for managing complex applications effectively. By integrating Grafana with Prometheus, Loki, and Tempo, you gain a comprehensive view of your system's performance, logs, and traces. This setup helps not only monitor but also debug and optimize your applications.

With Tracetest, you can further ensure the reliability of your system through proactive testing. As you saw today, implementing these observability practices allows for a more resilient and maintainable system, providing insights that are crucial for modern DevOps practices.

Frequently Asked Questions

Q. What are the stages of end-to-end testing?

End-to-end testing typically involves planning, where test cases and scenarios are defined; setup, which includes configuring the environment and integrating necessary tools; execution, where the test cases are run to simulate real-world user interactions; and validation, where results are analyzed to ensure the system behaves as expected. In the context of trace-based testing, this also includes inspecting traces to verify internal processes.

Q. Is Prometheus a visualization tool?

No, Prometheus is not a visualization tool. It's primarily a metrics storage and monitoring system. It collects, stores, and queries time-series data. However, Prometheus integrates well with visualization tools like Grafana, which can be used to create dashboards and visualizations for the metrics stored in Prometheus.

Q. What is "trace" in "trace-based testing"?

A trace in testing represents the journey of a request or transaction through various services in a system. In distributed systems, traces help track the flow of requests across different components, making it easier to identify bottlenecks, errors, or latency issues in the entire process.

Q. What is Grafana and Prometheus?

Grafana is an open-source visualization platform that creates interactive dashboards for monitoring metrics, logs, and traces. Prometheus, on the other hand, is a metrics collection and storage tool. Together, they form a powerful monitoring solution, with Prometheus gathering the data and Grafana visualizing it for better insights and troubleshooting.

Trace-based Testing With OpenTelemetry: Using Tracetest with OpenTelemetry

Adnan Rahić — Tue, 17 Sep 2024 10:29:07 +0000

Working with distributed systems can feel like a circus act. Trying to keep a dozen spinning plates in the air is hard—one slight misstep and everything can come crashing down.

Traditional testing methods are great for catching straightforward issues, but they often miss the nuanced problems that pop up when microservices start interacting with each other in unexpected ways. This is where trace-based testing comes in. By using distributed tracing in your observability tool, you’re not just checking if the end result is correct—you’re ensuring that every step in your application’s workflow is valid, from start to end.

This tutorial covers four things:

Basics of distributed tracing.
Differences between tracing and logging.
Introduction to OpenTelemetry, a set of observability APIs, SDKs, and tools used to instrument, generate, collect, and export telemetry data.
Introduction to Tracetest, a tool for trace-based testing.

Whether you work with microservices, cloud-native applications, mobile apps, serverless, or monolithic architectures, you’ll learn how to level up your testing strategy and ensure your systems perform at their best.

What is Distributed Tracing?

Distributed tracing maps how a single request moves through your application’s services.

Here’s a real-life example. When you request a product on an e-commerce platform, distributed tracing shows each step of the transaction, from the server’s HTTP request to database queries to the final page render. It’s crucial for understanding the system in real time, helping you spot bottlenecks or issues.

You might wonder—how is this different from logging? They both show what your system is doing. Their differences are key to effective monitoring and debugging.

Tracing vs Logging

Tracing provides a detailed journey of a request while logging captures specific moments. Logs are essential for debugging but don’t offer the whole picture. Tracing ties these moments together, creating a complete story that makes pinpointing the root cause of issues easier without sifting through scattered logs.

This detailed visibility offered by tracing naturally extends into testing strategies. Just as tracing connects the dots in monitoring, it also plays a critical role in testing workflows. This leads us to the comparison between traditional and trace-based testing, where the ability to see the entire process becomes invaluable.

Traditional Testing vs. Trace-based Testing

In traditional testing, you check if the output of a function or microservice matches the expected result. However, this can often miss issues like race conditions, performance bottlenecks, or incorrect service interactions.

Trace-based testing thoroughly validates workflows, ensuring not only the accuracy of the final output but also that each step is executed as intended. This is especially crucial in microservices environments, where interdependencies can make issues more difficult to detect.

Before we dive into how to leverage trace-based testing in your own systems, having the right tools at your disposal is crucial. That’s where OpenTelemetry comes into play. As one of the most widely adopted open-source projects, it provides a unified framework and software development kit (SDK) for collecting, processing, and exporting traces, metrics, and logs from your applications. With OpenTelemetry, you can easily instrument your code to generate the telemetry data you need to see how every core component of your system is performing and interacting.

Introduction to OpenTelemetry

OpenTelemetry is the open-source Swiss army knife for observability, providing tools to collect and export the three pillars of observability—traces, logs, and metrics from your applications. Whether you’re dealing with a sprawling microservices architecture or a monolithic app, OpenTelemetry integrates seamlessly, allowing you to monitor and troubleshoot easily.

Instrumentation in OpenTelemetry

Instrumentation is adding code to your application to generate telemetry data. With OpenTelemetry, you can instrument your services to automatically generate traces and metrics. This can be done manually by adding specific code or automatically by using OpenTelemetry's auto-instrumentation libraries, which inject the necessary tracing libraries into your application.

Let’s look at how to generate this telemetry data with OpenTelemetry using auto-instrumentation. We’ll be using a basic E-commerce application in Node.js for this tutorial. Create an Express server in app.js. Refer to this Express Guide to get started.

  const express = require('express');
  const { Pool } = require('pg');
  const app = express();
  const port = 5000;

  const pool = new Pool({
    user: 'postgres',
    host: 'localhost,'
    database: 'ecommerce',
    password: 'root',
    port: 5432,
  });

  // Routes
  app.get('/', (req, res) => {
    res.send('Welcome to the E-commerce App!');
  });

  app.get('/products', async (req, res) => {
    try {
      const result = await pool.query('SELECT * FROM products');
      res.json(result.rows);
    } catch (err) {
      res.status(500).send(err.message);
    }
  });

  // Start the server
  app.listen(port, () => {
    console.log(`Server running on http://localhost:${port}`);
  });

Run the server on your machine:

node app.js

The server will start running on port 5000:

Server running on http://localhost:5000

For this tutorial, I’ll showcase a basic Node.js server, but in your case, it can be a complex application in Java or a Kubernetes cluster with multiple microservices. It does not matter which tech stack you use since OpenTelemetry supports all the major programming languages like Java, .Net, Ruby, Python, Go, Rust, C++, and JavaScript.

Let’s enable the instrumentation in the server by creating the file instrumentation.js and adding the auto-instrumentation code to the server:

  const { NodeSDK } = require('@opentelemetry/sdk-node');
  const { ConsoleSpanExporter } = require('@opentelemetry/sdk-trace-node');
  const {
    getNodeAutoInstrumentations,
  } = require('@opentelemetry/auto-instrumentations-node');
  const {
    PeriodicExportingMetricReader,
    ConsoleMetricExporter,
  } = require('@opentelemetry/sdk-metrics');

  const sdk = new NodeSDK({
    traceExporter: new ConsoleSpanExporter(),
    metricReader: new PeriodicExportingMetricReader({
      exporter: new ConsoleMetricExporter(),
    }),
    instrumentations: [getNodeAutoInstrumentations()],
  });

  sdk.start();

Here’s a concise breakdown of the code above:

Import the necessary OpenTelemetry components for tracing and metrics.
Configure traceExporter, which uses ConsoleSpanExporter to print trace data to the console.
Configure metricReader, which uses PeriodicExportingMetricReader with ConsoleMetricExporter to print metrics to the console periodically.
Enable automatic instrumentation for standard Node.js modules for tracing.
Start the SDK using sdk.start() function, which initializes and begins collecting and exporting traces and metrics based on the configuration.

Install the required dependencies:

npm install express \
 @opentelemetry/sdk-node \
  @opentelemetry/api \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/sdk-metrics \
  @opentelemetry/sdk-trace-node \
  dotenv

Finally, run your application as you normally would, but use the --require flag to load the instrumentation before the application code.

node --require ./instrumentation.js app.js

After a while, you should see the spans and metrics printed in the console by ConsoleSpanExporter and ConsoleMetricExporter.

{
  resource: {
    attributes: {
      'service.name': 'unknown_service:C:/Program Files/nodejs/node.exe',
      'telemetry.sdk.language': 'nodejs',
      'telemetry.sdk.name': 'opentelemetry',
      'telemetry.sdk.version': '1.25.1',
      'process.pid': 17668,
      'process.executable.name': ' ',
      'process.executable.path': 'C:\\Program Files\\nodejs\\node.exe',
      'process.command_args': [
        'C:\\Program Files\\nodejs\\node.exe',
        '--require',
        './instrumentation.js',
        'C:\\Users\\Public\\Infrasity\\Tracetest\\e-commerce-app\\server\\app.js'
      ],
      'process.runtime.version': '20.10.0',
      'process.runtime.name': 'nodejs',
      'process.runtime.description': 'Node.js',
      'process.command': 'C:\\Users\\Public\\Infrasity\\Tracetest\\e-commerce-app\\server\\app.js',
      'process.owner': 'user',
      'host.name': 'MSI',
      'host.arch': 'amd64'
    }
  },
  traceId: '97975dbd81381c8b7bae28dafbff2bf8',
  parentId: undefined,
  traceState: undefined,
  name: 'fs statSync',
  id: '1a5eab1d5343483d',
  kind: 0,
  timestamp: 1724234720014000,
  duration: 128.8,
  attributes: {},
  status: { code: 0 },
  events: [],
  links: []
}

Now that we’ve set up tracing with OpenTelemetry, you’ve got a solid tool for capturing how your system operates, like seeing where requests slow down or fail. However, while OpenTelemetry shows you the data, it falls short in the following cases:

No Built-In Testing - OpenTelemetry provides traces but doesn’t test if your app functions as expected. It might show a slow request but won’t check its impact on the user experience.
Lack of Workflow Validation - It logs individual steps but doesn’t confirm if the entire process is correct, like logging an API call without verifying the follow-up actions.
No Assertion Validation - It records data but doesn’t check it against specific criteria. For example, it logs a database query but won’t validate the result.
Misses Workflow Impact - It identifies issues but doesn’t assess their impact on the overall system, such as detecting a delay without evaluating downstream effects.
No Automated Trace-based Testing - It captures trace data but doesn’t automate testing to verify if each trace meets expectations.

That’s where Tracetest steps in. It allows you to create tests based on the traces collected, validating not just the final outcomes but every step in the workflow. This helps you catch issues like incorrect service interactions or performance bottlenecks that might not be obvious with OpenTelemetry alone. By integrating Tracetest, you ensure your application operates smoothly in production, addressing potential problems before they impact users.

Using Tracetest for Trace-based Testing

Tracetest is a testing tool based on OpenTelemetry that allows you to test your distributed application. It will enable you to use data from distributed traces generated by OpenTelemetry to validate and assert if your application has the desired behavior defined by your test definitions.

How does Tracetest work?

In the architecture diagram above:

The Node.js server is instrumented to send traces to the OpenTelemetry Collector.
The Tracetest agent triggers tests and fetches the traces from the collector.
Finally, the agent synchronizes the test data back with Tracetest for analysis and validation.

This process enables continuous monitoring and testing of distributed systems using tracing data.

With a clear understanding of how Tracetest works, let's proceed to set it up in your system.

Setting up Tracetest

Sign up to app.tracetest.io and follow the get started docs to set up the Tracetest agent.
Create a new environment on the platform in the Environment tab.
Name your Environment and click on "Create."
After creating the environment, go to the Settings tab and follow the steps on the page to run the Tracetest Agent in your local environment.

tracetest start --api-key <your-api-key> --environment <your-env>

Version: v1.4.3

Tracetest start launches a lightweight agent. It enables you to run tests and collect traces with Tracetest.
Once started, Tracetest Agent exposes OTLP ports 4317 and 4318 to ingest traces via gRCP and HTTP.

 INFO  Running in desktop mode...
 INFO  Starting Agent with name MSI...
Agent is started! Leave the terminal open so tests can be run and traces gathered from this environment.
You can:
> Open Tracetest in a browser to this environment
  (Experimental) Open Dashboard
  Stop this agent

Now that your Tracetest Agent is successfully running, it can fetch all the traces generated by your services and sync them with Tracetest. Let’s start creating actual tests on Tracetest using these traces!

Trace-based Testing in a Node.js application using Tracetest

Let us use the same app.js we created when learning instrumentation in OpenTelemetry.

  const express = require('express');
  const { Pool } = require('pg');
  const app = express();
  const port = 5000;

  const pool = new Pool({
    user: 'postgres',
    host: 'localhost',
    database: 'ecommerce',
    password: 'root',
    port: 5432,
  });

  // Routes
  app.get('/', (req, res) => {
    res.send('Welcome to the E-commerce App!');
  });

  app.get('/products', async (req, res) => {
    try {
      const result = await pool.query('SELECT * FROM products');
      res.json(result.rows);
    } catch (err) {
      res.status(500).send(err.message);
    }
  });

  // Start the server
  app.listen(port, () => {
    console.log(`Server running on http://localhost:${port}`);
  });

Change instrumentation.js with the following code:

  const opentelemetry = require('@opentelemetry/sdk-node')
  const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node')
  const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');

  const dotenv = require("dotenv")
  dotenv.config()

  const sdk = new opentelemetry.NodeSDK({
    // OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is passed into "new OTLPTraceExporter" automatically
    traceExporter: new OTLPTraceExporter(),
    instrumentations: [getNodeAutoInstrumentations()],
    serviceName: 'quick-start-nodejs',
  })
  sdk.start()

Create a .env file. Get your Tracetest token and environment ID and add the following environment variables:

  # Get the required information here: https://app.tracetest.io/retrieve-token
  TRACETEST_TOKEN="<your-tracetest-token>"
  TRACETEST_ENVIRONMENT_ID="<your-tracetest-env-id>"

  # GRPC
  OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://localhost:4317/"

  # or, use HTTP
  # OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://tracetest-agent:4318/v1/traces"

Finally, run your application as you normally would, but use the --require flag to load the instrumentation before the application code.

node --require ./instrumentation.js app.js

The code above will automatically push the traces generated by OpenTelemetry auto-instrumentation to http://localhost:4317/, where the Tracetest Agent is running.

Ensure you run the Node.js application and the Tracetest Agent in the same network so the Tracetest Agent can ingest traces successfully.

Now, return to Tracetest to validate and test the application with distributed traces. In the “Run your first test” section, choose the trigger as HTTP and click on continue.

In the next screen, enter the URL of your server and click on “Run.”

When you go to the "Trace" tab in Tracetest, you’ll see a list of spans, which are individual units of work within a trace. Each span represents a specific operation or step in your application's workflow, such as an HTTP request, a database query, or a function execution.

Tracetest simplifies the view of these spans by organizing them in a clear, hierarchical structure, making it easy to follow the sequence of operations.

Switch to the "Test" tab to start adding tests on every trace span. Click the "Add Test Spec" button.

Let’s add a test to validate the server's response. Start by selecting the "Tracetest trigger" span, which represents the initial action in your test. In the "Add Assertions" section, choose the attribute attr.response.body. This allows you to check specific details of the server's response. Enter the expected value, which in our case is [], since the server returns an empty array.

Give the test spec a name and save it. Once saved, Tracetest will apply this spec to the test run whenever you trigger it, checking if the application behaves as expected according to the criteria you’ve set.

Run the test again and verify if the test spec passed or failed.

Similarly, let's create tests for an API. Select an API trace and add assertions to verify key aspects of the API's behavior. For example, you might check that the method used is "GET". You can also validate the route to ensure it contains the expected data or matches a specific format. By setting these assertions, you ensure that the API behaves as intended and meets your functional requirements.

Save the test spec and run tests again:

In the above image, you can see that one of the assertions failed the test because the expected value localhost:5001 does not match the actual value localhost:5000.

Let’s create assertions for database connection next.

Now, add tests for database queries by selecting the trace that corresponds to your database operations. Focus on specific spans related to the database interactions, such as queries or transactions.

Run the test after creating all the test specs to know how many tests pass and how many fail.

You have successfully tested the application in Tracetest using OpenTelemetry traces.

Wrapping Up: Trace-Based Testing for Easier Validation

Today, you learned about how trace-based testing with OpenTelemetry and Tracetest can simplify how you validate complex systems. Tracing offers a complete view of a request’s journey, far beyond what logs can provide. By adding Tracetest to your workflow, you’re not just monitoring—you’re testing every step of your application’s process.

While adopting trace-based testing might seem like a shift, the advantages are huge. With Tracetest, you’re not only ensuring system reliability but also gaining confidence that everything works as intended before hitting production.

Frequently Asked Questions

What is OpenTelemetry used for?

OpenTelemetry is an open-source observability framework used to collect, process, and export telemetry data (traces, metrics, and logs) from applications. It helps developers and operations teams monitor the performance of distributed systems, troubleshoot issues, and gain insights into how their applications are functioning in real time.

What is the difference between OpenTelemetry and Prometheus?

OpenTelemetry and Prometheus serve different but complementary purposes in observability. OpenTelemetry is focused on collecting and exporting traces, metrics, and logs from applications, providing a unified way to instrument and observe systems.
On the other hand, Prometheus is a monitoring and alerting tool designed explicitly for metrics collection and querying. While OpenTelemetry can generate and export metrics, Prometheus specializes in scraping and storing these metrics for monitoring purposes.
They can be used together, with OpenTelemetry generating the metrics and Prometheus storing and analyzing them.

What is the “trace” in trace-based testing?

In testing, a trace refers to a record of the execution path of a request or transaction as it moves through different components of a system. Tracing helps in understanding the flow of data, identifying bottlenecks, and ensuring that each step in the workflow is executed correctly. In trace-based testing, these traces are used to validate that not only the final output but each part of the system’s process works as intended.

Try Tracetest Yourself!

Last, but not least, do you want to learn more about Tracetest and what it brings to the table? Check the docs and try it out by signing up today!

Also, please feel free to join our Slack community, give Tracetest a star on GitHub, or schedule a time to chat 1:1.

Trace-Based Tests with GraphQL in Action!

Adnan Rahić — Tue, 10 Sep 2024 11:42:57 +0000

Trace-based Tests with GraphQL in Action 💣

Join us for the GraphQL and Playwright test trigger webinar on September 11th!

GraphQL is a query language for APIs and a runtime for fulfilling those queries with your existing data. It offers a clear, comprehensive description of your API's data, allowing clients to request exactly what they need—nothing more, nothing less. This approach simplifies API evolution and enables powerful developer tools.

Wouldn't it be great if Tracetest had a GraphQL trigger for testing GraphQL APIs?

Our community certainly thinks so. In fact, three community members have requested this feature in recent months.

Introducing the Tracetest GraphQL Trigger 🔥

Great news! The team has just released a brand-new GraphQL trigger type.

Join our demo environment to try it yourself!
Or, check out the guided example to run it in your own environment.

You can now effortlessly test GraphQL APIs using OpenTelemetry and trace-based testing. This feature is a natural addition to your existing HTTP and gRPC API tests.

This update enables you to run tests using GraphQL queries, expanding your API testing capabilities.

You have two options: upload your own schema or use schema introspection. With the latter, Tracetest automatically fetches the schema from your API URL.

From here, you can create test specs just as you would with other triggers. To streamline the creation and execution process and take advantage of any supported integrations. Navigate to the Automate tab to download or copy the definition file and the trigger script.

Testing GraphQL APIs

Running trace-based tests against GraphQL APIs has never been simpler. With this new trigger you can finally get the same test coverage for your GraphQL APIs as for your existing HTTP and gRPC APIs.

There are 4 steps to create and run a GraphQL trigger test:

Create a test with the GraphQL trigger 🕵️
Add your GraphQL API endpoint 🔗
Use the GraphQL Schema Introspection 🔍
Add a GraphQL query ❓
Hit run 🏃
Add test specs 🧪
Profit 🤑

Let me walk you through it all step-by-step.

To run a test with the GraphQL trigger, click the GraphQL trigger type when creating a new test.

Add your GraphQL API endpoint. Let’s use the Pokeshop demo as a sample.

https://demo-pokeshop.tracetest.io/graphql

Click the Use GraphQL Introspection button on the Schema tab to load the GraphQL schema.

Go back to the Query tab. Let’s add a mutation with two fields.

mutation create {
  createPokemon(name: "Charizard", type: "flying", isFeatured: false, imageUrl: "https://img.pokemondb.net/artwork/large/charizard.jpg") {
    id
    name
    isFeatured
    type
    imageUrl
  }

  importPokemon(id: 38) {
    id
  }
}

This will trigger the test and show you a trace response to start building test specs. The kicker here is that the two fields in the mutation will show two flows in the trace response.

You can see both the create and import flow in the trace.

To validate the GraphQL API, you can create a test spec like this:

- selector: span[tracetest.span.type="http" name="POST /graphql" http.method="POST"]
  name: Validate route is "/graphql" and data is valid.
  assertions:
    - attr:http.route   =   "/graphql"
    - attr:http.response.body =
      '{"data":{"createPokemon":{"id":224318,"name":"Charizard","isFeatured":false,"type":"flying","imageUrl":"https://img.pokemondb.net/artwork/large/charizard.jpg"},"importPokemon":{"id":38}}}'

You can also create specific test specs for the async flows in the Pokeshop demo like validating the RabbitMQ queue and the async API triggered behind the RabbitMQ queue.

But, what’s more exciting is that you can validate database interactions. In this sample you can validate both the import and create database calls in the same test. Here’s the validation for the create flow.

And, by selecting the database span below the RabbitMQ queue, you can validate the import flow as well.

Just like that, in a few minutes, you’ve created a GraphQL API test that validates two flows with multiple database interactions, async APIs and a message queue. If that isn’t magic, I don’t know what it. 🪄

GraphQL API Trigger with the Playwright Engine and Test Observability

With Tracetest, you can use the new GraphQL trigger alongside any other feature. Whether you want Tests as part of a Test Suite, plan to use them with the Synthetic Monitor Framework, or aim to include them in GitHub Actions—it's all possible.

A game-changing feature would be combining tests using both the GraphQL trigger and Playwright engine trigger into a Run Group. Then, add these to a Synthetic Monitor and run them on a schedule to continuously validate your applications in real time!

Try This in Tracetest yourself!

Check out the GraphQL trigger documentation to get started with your own tests. We're eager to hear your thoughts on this first version of the GraphQL Trigger!

Last, but not least, do you want to learn more about Tracetest and what it brings to the table? Check the docs and try it out by signing up today!

Also, please feel free to join our Slack community, give Tracetest a star on GitHub, or schedule a time to chat 1:1.

Announcing Tracetest Enterprise On-Prem Solution

Adnan Rahić — Mon, 19 Aug 2024 14:30:48 +0000

We are thrilled to announce the release of the on-prem enterprise self-hosted Tracetest! You can now get access to our cloud-based managed Tracetest experience in your own infrastructure. This new offering provides the same robust features and user-friendly experience as our cloud-based solution, now tailored for enterprises that require on-prem deployment.

Self-Hosted Tracetest: Ultimate Control and Customization

Our SaaS solution has empowered countless developers with seamless and efficient trace-based testing. However, we understand that organizations have specific compliance, security, and performance needs that sometimes require on-prem solutions.

The on-prem enterprise version addresses these requirements by providing:

Enhanced Security: Your data remains within your organization's infrastructure, providing an extra layer of security and compliance with internal policies and regulations. This is crucial for organizations in highly regulated industries or those with specific data residency requirement
Customization Options: Tailor the software to meet your specific requirements. Set up deep customization to suit your unique needs, whether it's integrating with legacy systems, tweaking the testing environment, or configuring specific security protocols.
Control Over Updates: Manage when and how updates are applied.

Which Tracetest Version is Right for You?

Choosing between the Managed Cloud and the Self-Hosted On-Prem version of Tracetest comes down to your organization's specific needs:

Go with Managed Cloud if you prioritize ease of use, rapid scalability, and prefer not to manage infrastructure or updates. This option is perfect for organizations looking for a seamless, low-maintenance solution that can quickly adapt to changing needs.
Opt for Self-Hosted On-Prem if you require full control over your environment, need to comply with strict data regulations, or want to deeply customize your testing setup. This version is ideal for large or security-focused organizations with unique needs.

Getting Started

If you're ready to take control of your testing infrastructure with the On-Prem version of Tracetest, we're here to help you every step of the way. We're committed to making your onboarding to the on-prem enterprise version as smooth as possible! We’ll create a dedicated Slack channel and assign a support team to assist you with deployment, configuration, PoC, and ongoing support to ensure that you are up and running as fast as possible.

Please reach out to our team for a personalized demo and pricing information or contact our founder Ken at ken@kubeshop.io.

Tracetest Monitors: Trace-based testing meets Synthetic Monitoring 🔥

Adnan Rahić — Wed, 26 Jun 2024 11:42:05 +0000

Are you ready to get your mind blown!? 🧠💥

Trace-based synthetic monitoring is here! You can now create “Monitors” for Tracetest tests and test suites.

Tracetest Monitors is a framework for creating scheduled runs of tests and test suites, and getting alerted when they fail. With native support for webhooks, you can choose how to integrate with you favorite alerting tools!

We’re dedicated to user experience and giving customers the easiest way of automating test runs without needing external CI tools. We want to give you the best possible tooling to embrace the “test in production” mindset too! Let’s be honest, we all do it! 😎

Join our demo environment to try it out yourself!!

The Problem — Scheduled Testing is Not as Simple as it Sounds

Test observability and trace-based testing has become a staple in the OpenTelemetry community. Having the power of distributed tracing at your fingertips when troubleshooting tests and writing assertions is immense. It’s highly advocated by the OpenTelemetry contributors in the official demo repository. While they, and all our other customers, are rolling their own automation with CI, to get the full benefit of trace-based testing, we’ve noticed it can often be a nuisance.

Configuring automated trace-based testing across multiple environments, using several tests, with a set timeframe of, let’s say, every 15 minutes, is not as simple as it sounds. You’d “just” need a CI tool of sorts, like GitHub Actions, or Jenkins. You’d maybe even settle for “just” creating a Kubernetes Job and defining it to run as a cron.

All of this is not as simple as using the word “just”.

Let’s normalize that “just” is relative and can’t apply the same to me, you, or someone else.

DevOps work like this can get exponentially more complex the more moving parts you introduce. A lot of the time SREs can get overloaded. That’s why shifting left and enabling the entire engineering team to perform synthetic testing in production, and across all your environments, is so powerful.

The Idea — The Beginning of Synthetic Testing

We introduced the Automate tab back in July of 2023 including this guide about simulating synthetic monitoring with GitHub Actions.

Making trace-based synthetic monitoring a reality has been a dream since then. But, waking up from dreams takes you back to reality. Back in July of last year we were hyper-focused on releasing Tracetest Open Beta and making Tracetest widely available for our users as a cloud-based managed platform.

We’ve grown and matured since then. Finally, reaching a point where adding scheduled runs with Monitors has become reality!

The Solution — Trace-based Synthetic Monitoring

The first-ever native synthetic monitoring tool for trace-based testing is live! Create a Monitor to run tests and test suites on a schedule.

Get alerted via webhooks by integrating with your favorite alerting tools.

Monitors leverage the existing Runs and Run Groups resources in Tracetest and build on top of it by enabling scheduling and alerting. Every Monitor you define will be presented as a Run Group with an additional tag using the specific name of the Monitor. Here you can see a list of Run Groups that include two Run Groups labeled as a Monitor.

Selecting the #c85ea8f8-805d-4064-86ea-1aad2074e9c9 Run Group shows which tests are part of the Monitor.

Filtering Runs by the Monitor tag is also available in the Runs view.

Releasing Monitors as a Synthetic Monitoring feature has been a long-standing dream, and it has finally arrived. This is the culmination of almost a year of planning and laying the groundwork to launch it for you! Now you can finally get the full benefit of test observability and trace-based testing and easy-to-use test automation for both production and pre-production environments.

How to Start Using Tracetest Synthetic Monitoring?

Make sure to use Tracetest v.1.3.1 and above. There are no other requirements. Click on the Monitors tab, click Create and have fun!

What’s Next?

First and foremost, we welcome your feedback on the initial version of Tracetest Monitors!

We recognize that there are big opportunities for improvement. Ensuring that the features we're developing meet the community's needs is our priority!

Last, but not least, do you want to learn more about Tracetest and what it brings to the table? Check the docs and try it out by signing up it today!

Also, please feel free to join our Slack community, give Tracetest a star on GitHub, or schedule a time to chat 1:1.

Using Env Vars to Include & Exclude OpenTelemetry Node.js Libraries

Adnan Rahić — Tue, 11 Jun 2024 11:58:07 +0000

OpenTelemetry's auto-instrumentation for Node.js is the open standard for tracing your applications. The default setting includes a wide range of events, some of which might not be meaningful or relevant to you. This can create a lot of noise in your distributed traces, making it harder to identify the insights that matter.

To run Node.js with OpenTelemetry auto instrumentation, you'll need to install the Node.js auto instrumentation module.

npm install --save @opentelemetry/api
npm install --save @opentelemetry/auto-instrumentations-node

Enable auto instrumentation by requiring this module using the --require flag:

node --require '@opentelemetry/auto-instrumentations-node/register' app.js

If your Node application is encapsulated in a complex run script, you can also set it via an environment variable before running Node.

env NODE_OPTIONS="--require @opentelemetry/auto-instrumentations-node/register"

The module is highly configurable using environment variables. Many aspects of the auto instrumentation's behavior can be configured for your needs. But, it’s sometimes hard to understand which instrumentations are most useful for your application. This might include HTTP events, database events, or other types of events depending on the nature of your application.

For instance, file system (fs) events and TCP (net) events are included in the auto-instrumentation, but these might not be useful in every application. Luckily, OpenTelemetry provides ways to customize the instrumentation to suit your needs.

Enable/Disable Auto Instrumentations with Env Vars

One common problem is the inclusion of file system spans, which might not be relevant to all applications. Disabling these can help to streamline your data and make it easier to interpret.

By default, all supported Instrumentations are enabled. With the OTEL_NODE_ENABLED_INSTRUMENTATIONS environment variable, you can enable certain instrumentations by providing a comma-separated list of the packages without the @opentelemetry/instrumentation- prefix.

To enable only @opentelemetry/instrumentation-http and @opentelemetry/instrumentation-express you can run this command below.

export OTEL_NODE_ENABLED_INSTRUMENTATIONS="http,express"

Enable/Disable Auto Instrumentations Programmatically

You can also initialize OpenTelemetry and configure the auto instrumentations programmatically.

Custom configuration for each auto instrumentations package can be passed to the getNodeAutoInstrumentations function. You provide an object with the name of the instrumentation as a key and its configuration as the value.

const opentelemetry = require("@opentelemetry/sdk-node")
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node")
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc')
const { Resource } = require("@opentelemetry/resources")
const { SemanticResourceAttributes } = require("@opentelemetry/semantic-conventions")
const { NodeTracerProvider } = require("@opentelemetry/sdk-trace-node")
const { BatchSpanProcessor } = require("@opentelemetry/sdk-trace-base")

const dotenv = require("dotenv")
dotenv.config()

const resource = Resource.default().merge(
  new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: "quick-start-nodejs-manual-instrumentation",
    [SemanticResourceAttributes.SERVICE_VERSION]: "0.0.1",
  })
)

const provider = new NodeTracerProvider({ resource: resource })
const exporter = new OTLPTraceExporter()
const processor = new BatchSpanProcessor(exporter)
provider.addSpanProcessor(processor)
provider.register()

const sdk = new opentelemetry.NodeSDK({
  traceExporter: exporter,
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-fs': {
        enabled: false
      },
      '@opentelemetry/instrumentation-net': {
        enabled: false
      },
    })
  ],
  serviceName: 'quick-start-nodejs-manual-instrumentation'
})
sdk.start()

Check out this example in a code sample, here on GitHub. There’s a runnable sample app you can start with Docker Compose to see it for yourself. Here’s what the trace will look like without the noisy fs and net events.

For more info you can also refer to the OpenTelemetry docs. It includes a section on excluding certain instrumentation libraries.

Key Takeaways

When using Node.js with OpenTelemetry auto instrumentation, you need to understand which auto instrumentations are most useful for your application. This might include HTTP events, database events, or other types of events that you need to make your app reliable.

You can enable specific instrumentations from environment variables, but also configure them programmatically. Other options you get by default include setting the OTLP endpoint, headers, and much more. See the list below.

export OTEL_TRACES_EXPORTER="otlp"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_COMPRESSION="gzip"
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://your-endpoint"
export OTEL_EXPORTER_OTLP_HEADERS="x-api-key=your-api-key"
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="x-api-key=your-api-key"
export OTEL_RESOURCE_ATTRIBUTES="service.namespace=my-namespace"
export OTEL_NODE_RESOURCE_DETECTORS="env,host,os,serviceinstance"
export OTEL_NODE_ENABLED_INSTRUMENTATIONS="http,express"
export OTEL_SERVICE_NAME="client"
export NODE_OPTIONS="--require @opentelemetry/auto-instrumentations-node/register"
node app.js

In conclusion, the Node.js OpenTelemetry auto instrumentation is a lifesaver and includes a lot of useful instrumentations by default. However, it’s up to you to customize the instrumentations to suit your application. By enabling only relevant instrumentations, you can reduce the noise in your tracing, making it easier to gain valuable insights.

Interested in proactively using traces instead of just reactively troubleshooting in production? Check out the Tracetest docs and give it a try by signing up today. Tracetest enables you to add test observability to all your existing tests. It integrates with Playwright, Cypress, k6, Artillery, and can run tests against APIs (HTTP/gRPC), message queues like Kafka, and much more!

Also, please feel free to join our Slack Community, give Tracetest a star on GitHub, or schedule a time to chat 1:1.

Tracetest + Artillery Launch Week Recap 💥

Adnan Rahić — Fri, 19 Apr 2024 15:13:50 +0000

This week was Tracetest’s first-ever Launch Week. We’ve been working on a major integration with Artillery for the last month and our team is beyond excited to share it with you all!

What’s a launch week?

Each day this week, we’ve published new educational content on how to quickly get started with trace-based performance testing. This included docs guides, video tutorials, code samples, a blog post, and a webinar with live demos.

We’re not the first company to run a launch week. Shoutout to Supabase for starting the hype around launch weeks. However, they’re a great way to give a huge new product update the time in the spotlight it deserves. And, give both the Artillery and Tracetest teams praise for releasing amazing new features!

What happened this week?

We shared daily updates on our LinkedIn and Twitter pages, as well as in our community. In addition, we collaborated with our friends from Artillery on a blog post and live stream, complete with live demos. Lastly, we compiled this recap, making it easier for you to catch up on everything in one place.

Without further ado, here’s what we did this week.

Monday - Day 1: Artillery + Tracetest Integration Quick Starts

We've announced a quick start guide for using the Artillery and Tracetest integration, including a runnable code example that you can start using in seconds!

Additionally, there's a video tutorial by Oscar that provides a step-by-step guide on how to use the integration.

‍View the full video on YouTube, here.

Tuesday - Day 2: Artillery + Tracetest Integration Announcement

We were excited to see our friends at Artillery publish the official integration announcement in their blog.

Read the full blog post, here.

Wednesday - Day 3: Performance Testing with Distributed Tracing using Artillery, Playwright, and Tracetest

Day 3 was a banger! We sat down with our friends Bernardo and Ines from Artillery and discussed everything from performance testing to using distributed tracing with OpenTelemetry for end-to-end testing. Oscar and Ines covered hands-on code demos of the Artillery + Tracetest integration, but also demoed how to use Playwright too!

‍View the full video on YouTube, here.

Thursday - Day 4: Enhancing the Artillery Playwright engine with Tracetest for observability-driven testing

Day 4 was packed with guides and examples from the webinar collaboration! It includes a docs quick start and example code for using the Artillery + Playwright + Tracetest for trace-based performance testing with end-to-end visibility.

Friday - Day 5: Recap‍

Today is the last day of the Launch Week. We’re dedicating today to a recap of everything the community has learned during the week and compiling all learning resources in one place.

We’re so proud to have had the chance to collaborate with the lovely folks at Artillery. Thanks to Ines, Bernardo, and Hassy for their hard work and awesome presentation skills!

What's next?

Would you like to learn more about what Tracetest and Artillery can solve?

Try out Artillery

Follow this guide to try out running an example performance test.
Check out this example to use Artillery Cloud.

Try out Tracetest

Check the guides in the docs.
Try it out by signing up today.

Also, please feel free to join our Slack Community, give Tracetest a star on GitHub, or schedule a time to chat 1:1.

Observability at KubeCon + CloudNativeCon Europe 2024 in Paris

Adnan Rahić — Tue, 26 Mar 2024 14:58:45 +0000

Greetings from KubeCon + CloudNativeCon Europe! It was a wild and awesome ride last week! Ken and I had a blast at both the Co-Located Observability Day on Tuesday, as well as during the main conference! I have a lot to share. Here’s a recap of how the observability community has grown in Europe since last year.

KubeCon + CloudNativeCon Venue

KubeCon + CloudNativeCon Europe 2024 was in Paris the week of March 18th. I thought it would be interesting to recap some of the events from the Tracetest team’s perspective with an eye towards Observability.

First, the venue Paris Expo Porte de Versailles, was beautiful. Our team had a booth on the main floor, but also attended the co-located Observability Day event prior to the main conference.

The Observability Day got two rooms this year! Finally! Here’s what the main room looked like at 8:30 before it got packed to the brim when the talks started. Absolutely massive!

Observability Day and OpenTelemetry

Let me start off by discussing Observability Day. The agenda was packed with informative sessions and discussions that shed light on the latest trends and advancements in observability.

The CNCF finally provided a second room to have more sessions and accommodate the number of interested people!

Opening session by OpenTelemetry community committee members

Austin Parker, Honeycomb, started off the co-located event. He covered OpenTelemetry project updates and future goals.

Most notable were updates to:

Stability
Events & RUM
Continuous profiling

Since last Observability Day 2023 in Chicago, 4 more languages have been marked stable!

Events will be optimized for real-user monitoring!

Profiling will be added this year!

But the biggest bombshell is that OpenTelemetry has applied for graduation in the CNCF.

Eduardo Silva, Chronosphere, then took the stage to talk about updates happening in the world of FluentBit.

He covered latest updates and the roadmap for 2024.

Pavol Loffay (Red Hat) and Jonah Kowall (Aiven) covered exciting updates to Jaeger.

The roadmap looks awesome!

As do the v2 goals!

This was a great segue into the more technical presentations of the day.

We did catch all the sessions at Observability Day, but I want to share some of my favorites with you.

Hey OpenTelemetry, where’s my error!?

Dude, Where’s My Error?: How OpenTelemetry Records Errors, and Why It Does It Like That — The session presented by Reese Lee, New Relic & Adriana Villela, ServiceNow.

They emphasized the importance of tracking errors and understanding system functionality for user satisfaction. They used a demo to showcase OTel's error logging, metadata-enhanced troubleshooting, and the difference between errors and exceptions. The session explained error visualization across different backends and the effect of different span kinds on error reporting.

How to think about overhead?

How to Think About Instrumentation Overhead — Jason Plumb (Splunk) did an amazing talk on instrumentation overhead in an observability context. Novice observability practitioners who were previously overly obsessed with performance had approached instrumentation with skepticism due to concerns about latency degradation or resource consumption.

The causes of overhead were covered, illustrating why it is difficult to measure and predict. Jason also showcased practical techniques for understanding overhead in one's environment and strategies for coping with it were presented.

FluentBit vs. OpenTelemetry Collector

Telemetry Showdown: Fluent Bit Vs. OpenTelemetry Collector - a Comprehensive Benchmark Analysis — Henrik Rexed, Dynatrace.

In a push to standardize observability practices, the cloud-native community has embraced OpenTelemetry, offering a unified framework for metrics, logs, and traces. In the past log processing relied on agents like Fluentd, evolving into FluentBit. With FluentBit's recent expansion to support additional signals and the rise of OpenTelemetry Collector, a question arises: "Which is the superior choice for performance?"

Henrik explained the performance differences, ease of use, compatibility, and more!

Improving Serverless monitoring at Google

Monitoring Serverless Workloads with OpenTelemetry and Prometheus - Ridwan Sharif, Google.

In this presentation Ridwan explained the challenge with observability and serverless platforms. To enable OpenTelemetry tracing and Prometheus-based metrics in Google's Cloud Run they had to implement new updates to sidecars in Cloud Run.

This enabled flushing push-based OpenTelemetry traces before containers were killed. But also scraping with Prometheus. This presentation explains how these improvements to Cloud Run enabled the use of open-source observability tooling to improve serverless observability.

What's a good OpenTelemetry mindset?

Shift Into an Observability Mindset with OpenTelemetry - Daniel Gomez Blanco, Skyscanner.

Daniel covered an amazing topic. Embracing observability in your organization.

To do so, you must shift your mindset to rethink monitoring and debugging practices. This means moving to OpenTelemetry and communicating the value of OpenTelemetry as the ideal tool for the job both today and moving forward. Daniel also covered adoption best practices as well. Can't wait to get the recording and re-watch this amazing talk.

How to build dynamic OpenTelemetry Collector configs?

OpAMP in Action: User Configurable Observability Pipelines - Srikanth Chekuri, SigNoz.

Observability pipelines using OpenTelemetry Collector can't be configured dynamically. This means you need to restart the collector when the config is changed. With OpAMP you can bypass this to build dynamic user configurable observability pipelines!

Observability Vendors at KubeCon

Moving on, let's explore the diverse range of observability vendors present at KubeCon in Paris this year. I counted 29 vendors who set up booths, each offering unique solutions and perspectives.

As promised, I have compiled a collection of photos to give you a glimpse into the vibrant atmosphere and impressive setups.

Check out the video on YouTube, here.

Interviewing OpenTelemetry Maintainers, Approvers, and Contributors

I had the chance to talk to OpenTelemetry community members about their experience and work in the community. The interviews include Juliano Costa (OpenTelemetry Demo maintainer), Kayla Reopelle (OpenTelemetry Ruby Approver), and Adriana Villela (OpenTelemetry Contributor).

View the video with OpenTelemetry Demo Maintainer Juliano Costa, here.

Stay tuned for a complete case study with the OpenTelemetry maintainers!

Fostering a Sense of Community

In the world of observability, it's super cool to see how vendors, even while competing, come together like old friends. It’s awesome to witness these bonds and collabs, giving the whole scene a cozy, tight-knit vibe.

Need proof? Check out how folks from ServiceNow and New Relic teamed up for sessions, not to mention how Google engineers are committed to open-source observability!

And then there’s us at Tracetest, playing it cool like Switzerland in the midst of OpenTelemetry. We're all about bridging gaps, working with everyone to bring trace-based testing into the spotlight. It's all about fostering that spirit of unity and teamwork!

Where do we go from now?

Now once we're all back from KubeCon in Paris, let’s continue the discussion and try contributing even more to the OpenTelemetry project. The OpenTelemetry project has filed for graduation in the CNCF.

Our team has been active in the last year, with numerous PRs, both code contributions and content for the OpenTelemetry blog. I am hoping we can ramp up even more and help the developer community embrace OpenTelemetry, making it easier than ever to get started.

What do you think?

I’ve shared my thoughts and experiences from KubeCon. Let me hear your ideas as well! What topics would you like to see covered? Feel free to reach out and share your suggestions!

Do you want us to contribute to something specific in the OpenTelemetry project?
Do you need a specific feature in Tracetest to help you work with OpenTelemetry?
There are no bad ideas!

I’m eager to hear your thoughts. Feel free to reach out to me any way you prefer! I am active on LinkedIn daily and happy to answer questions or just chat about OTel.

See you at KubeCon North America in Salt Lake City!

To wrap it up, KubeCon was a super exciting experience. The observability vendors showed off their skills, and the awesome sense of community was all about working together and contributing to OpenTelemetry.

If you enjoyed this post, share it with your friends!

Feel free to connect with us on social media, join our Slack community, and give us a ⭐ on GitHub.

Crafting Observable Cloudflare Workers with OpenTelemetry

Adnan Rahić — Wed, 14 Feb 2024 12:46:57 +0000

Serverless architectural patterns struggle with visibility. They’re difficult to troubleshoot in production and complex to test across development and staging environments including integration tests.

Today you’ll learn how to gain insight into your development lifecycle and test critical flows while building production-ready Cloudflare Workers that interact with a D1 database.

This includes adding:

Distributed tracing with OpenTelemetry for troubleshooting.
Trace-based testing with Tracetest for integration testing and testing staging/production deployments.

Why? So you can:

Test what you usually can’t! Apply assertions against the Cloudflare Worker runtime, external API calls, and a serverless database.
Troubleshoot failed tests, with traces. Use OpenTelemetry-based distributed traces.
Stop the blame game. With a view of the entire flow, from upstream API to database, quickly determine the cause of the failure.
Gain better observability! As tracing data from instrumentation is used to build trace-based tests, developers will want to add more insightful and meaningful instrumentation.

Once you’re done with the tutorial, you’ll have configured testing Cloudflare Workers in live staging and production deployments.

If you’re eager to start, clone the example from GitHub and get a Tracetest Agent public URL and Token after signing up at app.tracetest.io. Sign up for a Cloudflare account on dash.cloudflare.com/. Update the values in wrangler.toml, deploy your Cloudflare Worker, and run tests against deployed code! Read the quick start instructions, here.

git clone https://github.com/kubeshop/tracetest.git
cd tracetest/examples/testing-cloudflare-workers
# Install modules and npx if you haven't already
npm i npx -g
npm i
# Sign in to Cloudflare
npx wrangler login
# Set the <TRACETEST_URL> in wrangler.toml
npx wrangler d1 create testing-cloudflare-workers
# Set the <YOUR_DATABASE_ID> from the command above in wrangler.toml
npx wrangler d1 execute testing-cloudflare-workers --file=./schema.sql
# Deploy Cloudflare Worker
npm run deploy
# Run tests!
tracetest run test -f ./test/test-api.prod.yaml

What are Cloudflare Workers?

Cloudflare Workers are Cloudflare’s answer to AWS Lambda. They let you deploy serverless code instantly across the globe and are blazing fast. You write code and deploy it to cloud environments without the need for traditional infrastructure.

Install Cloudflare Dev Tools and Create the Boilerplate

There are three prerequisites to get started. You might’ve figured already but let me outline them below:

Sign up for a Cloudflare account.
Install npm .
Install Node.js.

You create a new Worker project with the create-cloudflare-cli also called C3. It’s a command-line tool designed to help you setup and deploy Workers to Cloudflare.

Open a terminal window and run C3 to create your Worker project.

npm create cloudflare@latest

This will prompt you to install C3, and lead you through setup. Let’s set up a basic worker.

Name your new Worker directory pokemon-api because I’ll demo how to fetch Pokemon from an external API.
Select "Hello World" script as the type of application you want to create.
Answer yes to using TypeScript.

You will be asked if you would like to deploy the project to Cloudflare. Go ahead and select yes.

You will be asked to authenticate, if not logged in already, and your project will be deployed to the Cloudflare global network.

Open your Cloudflare Dashboard and see that indeed you have deployed the Hello World Worker.

In your project directory, C3 has generated the following:

wrangler.toml: Your Wrangler configuration file. The Workers command-line interface, Wrangler, allows you to create, test, and deploy your Workers projects. C3 will install Wrangler in projects by default.
index.js (in /src): A minimal 'Hello World!' Worker written in ES module syntax.
package.json: A minimal Node dependencies configuration file.
package-lock.json: Refer to npm documentation on package-lock.json.
node_modules: Refer to npm documentation node_modules.

Create a Cloudflare D1 Database

The Cloudflare D1 Database is a serverless SQL database built on SQLite. It offers a native serverless architecture, a SQL-based dialect, and built-in JSON parsing and querying functions. With D1, you can easily deploy and maintain a database of any size, using a familiar query language and benefiting from features like point-in-time recovery and cost-effective pricing.

Start by creating a D1 database.

npx wrangler d1 create testing-cloudflare-workers

[Output]

✅ Successfully created DB 'testing-cloudflare-workers' in region EEUR
Created your database using D1's new storage backend. The new storage backend is not yet recommended for production workloads, but backs up your data via point-in-time
restore.

[[d1_databases]]
binding = "DB" # i.e. available in your Worker on env.DB
database_name = "testing-cloudflare-workers"
database_id = "<your_database_id>"

Add the d1_databases block to your wrangler.toml file.

# D1
[[d1_databases]]
binding = "DB" # i.e. available in your Worker on env.DB
database_name = "testing-cloudflare-workers"
database_id = "<your_database_id>"

Create a schema that defines a Pokemon table. Call it schema.sql and put it in the root directory.

DROP TABLE IF EXISTS Pokemon;
CREATE TABLE IF NOT EXISTS Pokemon (
  id INTEGER PRIMARY KEY,
  name TEXT,
  createdAt TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

Configure D1 locally and in your Cloudflare account.

# Local
npx wrangler d1 execute testing-cloudflare-workers --local --file=./schema.sql

# Deployed
npx wrangler d1 execute testing-cloudflare-workers --file=./schema.sql

Finally use the DB binding to query the database.

export interface Env {
  // If you set another name in wrangler.toml as the value for 'binding',
  // replace "DB" with the variable name you defined.
  DB: D1Database;
}

// ...

You’re ready to start building!

Use Wrangler CLI to Develop Workers

The Workers command-line interface, Wrangler, allows you to create, test, and deploy your Workers projects. C3 will install Wrangler in projects by default.

After you have created your first Worker, run the wrangler dev command in the project directory to start a local server for developing your Worker. This will allow you to test your Worker locally during development.

npx wrangler dev

[output]
 ⛅️ wrangler 3.22.1
-------------------
✔ Would you like to help improve Wrangler by sending usage metrics to Cloudflare? … no
Your choice has been saved in the following file: ../../../../Library/Preferences/.wrangler/metrics.json.

  You can override the user level setting for a project in `wrangler.toml`:

   - to disable sending metrics for a project: `send_metrics = false`
   - to enable sending metrics for a project: `send_metrics = true`
⎔ Starting local server...
[wrangler:inf] Ready on http://localhost:8787
[wrangler:inf] GET / 200 OK (5ms)
[wrangler:inf] GET /favicon.ico 200 OK (1ms)
...

If you have not used Wrangler before, it will try to open your web browser to authenticate with your Cloudflare account. If you have issues with this step or you do not have access to a browser interface, refer to the wrangler login documentation for more information.

You will now be able to go to http://localhost:8787 to see your Worker running. Any changes you make to your code will trigger a rebuild, and reloading the page will show you the up-to-date output of your Worker.

Cloudflare Worker Boilerplate Code

The src/index.ts file contains the Worker code. This file will be triggered when hitting the / endpoint of your Worker.

Think of it as a tiny Node.js server. You can do all kinds of cool things here. Create routers, listen for different HTTP methods like POST etc, fetch external APIs, store data in databases, and even trigger other Workers!

Open up the src/index.ts. You’ll see boilerplate code.

/**
 * Welcome to Cloudflare Workers! This is your first worker.
 *
 * - Run `npm run dev` in your terminal to start a development server
 * - Open a browser tab at http://localhost:8787/ to see your worker in action
 * - Run `npm run deploy` to publish your worker
 *
 * Learn more at https://developers.cloudflare.com/workers/
 */

export interface Env {
    // Example binding to KV. Learn more at https://developers.cloudflare.com/workers/runtime-apis/kv/
    // MY_KV_NAMESPACE: KVNamespace;
    //
    // Example binding to Durable Object. Learn more at https://developers.cloudflare.com/workers/runtime-apis/durable-objects/
    // MY_DURABLE_OBJECT: DurableObjectNamespace;
    //
    // Example binding to R2. Learn more at https://developers.cloudflare.com/workers/runtime-apis/r2/
    // MY_BUCKET: R2Bucket;
    //
    // Example binding to a Service. Learn more at https://developers.cloudflare.com/workers/runtime-apis/service-bindings/
    // MY_SERVICE: Fetcher;
    //
    // Example binding to a Queue. Learn more at https://developers.cloudflare.com/queues/javascript-apis/
    // MY_QUEUE: Queue;
}

export default {
    async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
        return new Response('Hello World!');
    },
};

Let’s edit it to instead create a Pokemon directory.

Create the Cloudflare Worker Code

You’ll create an import flow. Send the ID of a Pokemon to the Cloudflare Worker, it handles getting the Pokemon info from an external API and stores it in the D1 database.

The Cloudflare Worker you’ll create will be accessible at the URL http://localhost:8787/.

To return JSON data, you’ll use the return Response.json(...) method. Async/Await flows are enabled by default as well!

Here’s an example of a POST request with a GET request to an external API from within the Cloudflare Worker and then inserting the data into D1. It’s a common point-of-failure that is hard to troubleshoot and test.

You’ll configure the Worker to listen for a POST request on path /api/pokemon. And, add an optional query parameter called id.

export interface Env {
  DB: D1Database
}

export async function addPokemon(pokemon: any, env: Env) {
  return await env.DB.prepare(
    "INSERT INTO Pokemon (name) VALUES (?) RETURNING *"
  ).bind(pokemon.name).all()
}

export async function getPokemon(pokemon: any, env: Env) {
  return await env.DB.prepare(
    "SELECT * FROM Pokemon WHERE id = ?"
  ).bind(pokemon.id).all()
}

async function formatPokeApiResponse(response: any) {
  const { headers } = response
  const contentType = headers.get("content-type") || ""
  if (contentType.includes("application/json")) {
    const data = await response.json()
    const { name, id } = data
    return { name, id }
  }
  return response.text()
}

const handler = {
    async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
    try {
      const { pathname, searchParams } = new URL(request.url)

      // Import a Pokemon
      if (pathname === "/api/pokemon" && request.method === "POST") {
        const queryId = searchParams.get('id')
        const requestUrl = `https://pokeapi.co/api/v2/pokemon/${queryId || '6'}`
        const response = await fetch(requestUrl)
        const resPokemon = await formatPokeApiResponse(response)

        const addedPokemon = await addPokemon(resPokemon, env)
        return Response.json(addedPokemon)
      }

      return new Response("Hello Worker!")
    } catch (err) {
      return new Response(String(err))
    }
    },
}

export default handler

Go ahead and run the Worker.

npx wrangler dev

Send a POST request to http://localhost:8787/api/pokemon to see the response from your Cloudflare Worker.

curl -X POST "http://localhost:8787/api/pokemon?id=1"

{"success":true,"meta":{"served_by":"v3-prod","duration":0.4586,"changes":1,"last_row_id":2,"changed_db":true,"size_after":16384,"rows_read":1,"rows_written":2},"results":[{"id":2,"name":"bulbasaur","createdAt":"2024-02-06 17:54:52"}]}

To validate that the Pokemon has been added, you can also run this command to query the database.

npx wrangler d1 execute testing-cloudflare-workers --local --command="SELECT * FROM Pokemon"

Redeploy the Worker via Wrangler.

npx wrangler deploy

Preview your Worker at <YOUR_WORKER>.<YOUR_SUBDOMAIN>.workers.dev.

Configure Troubleshooting with OpenTelemetry and Distributed Tracing

OpenTelemetry libraries for Node.js are stable and include auto-instrumentation.

Luckily for you and me, Erwin van der Koogh ****wrote an awesome package for wrapping the OpenTelemetry libraries in Cloudflare Workers. It’s called otel-cf-workers and it is beautiful! 🤩 — Let’s be awesome to each other and give him as many ⭐ as possible on GitHub!

To be able to use the OpenTelemetry library at all you have to add the Node.js compatibility flag in your wrangler.toml file.

compatibility_flags = [ "nodejs_compat" ]

Now, install the node modules.

npm i \
  @opentelemetry/api \
  @microlabs/otel-cf-workers

You need the @opentelemetry/api module to create custom spans, while the @microlabs/otel-cf-workers module is the OpenTelemetry wrapper.

Next up, add OpenTelemetry to your code.

import { trace, SpanStatusCode } from '@opentelemetry/api'
import { instrument, ResolveConfigFn } from '@microlabs/otel-cf-workers'
const tracer = trace.getTracer('pokemon-api')

export interface Env {
  DB: D1Database
    TRACETEST_URL: string
}

export async function addPokemon(pokemon: any, env: Env) {
  return await env.DB.prepare(
    "INSERT INTO Pokemon (name) VALUES (?) RETURNING *"
  ).bind(pokemon.name).all()
}

export async function getPokemon(pokemon: any, env: Env) {
  return await env.DB.prepare(
    "SELECT * FROM Pokemon WHERE id = ?;"
  ).bind(pokemon.id).all();
}

async function formatPokeApiResponse(response: any) {
  const { headers } = response
  const contentType = headers.get("content-type") || ""
  if (contentType.includes("application/json")) {
    const data = await response.json()
    const { name, id } = data

    // Add manual instrumentation
    const span = trace.getActiveSpan()
    if(span) {
      span.setStatus({ code: SpanStatusCode.OK, message: String("Pokemon fetched successfully!") })
      span.setAttribute('pokemon.name', name)
      span.setAttribute('pokemon.id', id)
    }

    return { name, id }
  }
  return response.text()
}

const handler = {
    async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
    try {
      const { pathname, searchParams } = new URL(request.url)

      // Import a Pokemon
      if (pathname === "/api/pokemon" && request.method === "POST") {
        const queryId = searchParams.get('id')
        const requestUrl = `https://pokeapi.co/api/v2/pokemon/${queryId || '6'}`
        const response = await fetch(requestUrl)
        const resPokemon = await formatPokeApiResponse(response)

        // Add manual instrumentation
        return tracer.startActiveSpan('D1: Add Pokemon', async (span) => {
          const addedPokemon = await addPokemon(resPokemon, env)

          span.setStatus({ code: SpanStatusCode.OK, message: String("Pokemon added successfully!") })
          span.setAttribute('pokemon.name', String(addedPokemon?.results[0].name))
          span.end()

          return Response.json(addedPokemon)
        })
      }

      return new Response("Hello Worker!")
    } catch (err) {
      return new Response(String(err))
    }
    },
}

const config: ResolveConfigFn = (env: Env, _trigger) => {
  return {
    exporter: {
      url: env.TRACETEST_URL,
      headers: { },
    },
        service: { name: 'pokemon-api' },
    }
}

export default instrument(handler, config)

Let me explain what’s happening here:

You’re importing the @opentelemetry/api and @microlabs/otel-cf-workers modules to enable OpenTelemetry tracing.
The @microlabs/otel-cf-workers module contains the instrument and ResolveConfigFn functions. You’ll use them to wrap the Cloudflare Worker and automatically generate traces.
The const tracer = trace.getTracer('pokemon-api') will instantiate a tracer object for you to create new spans in your code. You’ll do this to create a trace for the D1 insert operation. As you can see in the handler I’ve now wrapped the addPokemon function with a tracer.startActiveSpan('D1: Add Pokemon'...) trace span.
The formatPokeApiResponse function now contains manual instrumentation to add span attributes for the external API request. This is to validate the external Pokemon API works as expected.
The url: env.TRACETEST_URL in the ResolveConfigFn function sets the Tracetest Agent URL where you send traces to. I’ll walk you through creating tests further down where you’ll also add environment variables.

By configuring your Cloudflare Workers with the following settings, you can enable production observability and emit distributed traces. But what if you could also incorporate testing?

The Magic of Trace-based Testing for Serverless Architectures

Testing serverless architectures has long been a challenge due to limited visibility. However, observability through distributed traces is now providing a solution to this problem.

Trace-based testing takes uses existing OpenTelemetry traces as test specifications. This allows you to validate the behavior and performance of your distributed services and serverless functions.

Tracetest is a trace-based testing tool for building integration tests in minutes using OpenTelemetry traces. You can build test specs against trace data at every point of a request transaction.

To get started with Tracetest:

You’ll need to download the CLI for your operating system.
And, sign up for an account. Go ahead and do that now.

The CLI is bundled with Tracetest Agent that triggers your application and collects responses and traces for new tests. Learn more in the docs here.

Let me walk you through creating tests across your staging and production deployments.

Testing Cloudflare Workers in Staging and Production

Since I want to keep environments separate I’ll use the environments feature in Wrangler.

Create a new environment in Tracetest. Select to run the Tracetest Agent in the cloud.

OpenTelemetry will be selected as the default tracing backend. You’ll find the OTLP endpoint to send traces to.

Copy the HTTP URL and paste it in the wrangler.toml using a new section called [env.prod] . Make sure to append v1/traces to the end of the Tracetest URL. Make sure to use the database_id that you generated at the beginning of the tutorial.

# Production
[env.prod]
name = "pokemon-api"
main = "src/index.ts"
compatibility_date = "2023-12-18"
compatibility_flags = [ "nodejs_compat" ]
workers_dev = true
d1_databases = [
  { binding = "DB", database_name = "testing-cloudflare-workers", database_id = "<YOUR_DATABASE_ID>" },
]
[env.prod.vars]
TRACETEST_URL = "https://<YOUR_TRACETEST_AGENT_URL>.tracetest.io:443/v1/traces"

The TRACETEST_URL here is where the Tracetest Agent is running. Currently, in the cloud in your Tracetest account. To reference it in your Cloudflare Worker you define it in the interface Env and again set it in the exporter section.

// [...]
export interface Env {
    TRACETEST_URL: string
    // [...]
}

// [...]

const config: ResolveConfigFn = (env: Env, _trigger) => {
    return {
    exporter: {
      url: env.TRACETEST_URL,
      headers: { },
    },
    // [...]
    }
}

Deploy the Cloudflare Worker to the prod environment.

npx wrangler deploy --env prod

[Output]
 ⛅️ wrangler 3.27.0
-------------------
Your worker has access to the following bindings:
- D1 Databases:
  - DB: testing-cloudflare-workers (<YOUR_DATABASE_ID>)
- Vars:
  - TRACETEST_URL: "https://agent-(redacted)..."
Total Upload: 169.02 KiB / gzip: 38.56 KiB
Uploaded pokemon-api (3.79 sec)
Published pokemon-api (1.72 sec)
  https://pokemon-api.<YOUR_ACCOUNT>.workers.dev
Current Deployment ID: 6e5333b9-29de-4a83-84c5-dc582218bdba

Make sure you see the TRACETEST_URL environment variable in your Cloudflare account.

Move back to Tracetest and use the Cloudflare Worker production URL to trigger a test.

Switch to the Trace tab to see the full preview of the distributed trace.

From here you can add test specs to validate that the external HTTP request does not fail and that the D1 database import worked as expected. You can also check for cold starts!

Click the Test tab and add some test specs. First create a test spec to validate the function invocation was not a cold start.

Then make sure the external API request always returns status 200.

Finally, validate that the Pokemon that was added to the D1 database matched what the external API fetched.

Save the test specs.

You now see all test specs passing since the external HTTP request is valid, the invocation was not a cold start, and the Pokemon name matches what you were expecting!

All this enabled by OpenTelemetry tracing and Tracetest! What’s also awesome is that these tests are stored in your Tracetest account and you can revisit them and run the same tests again every time you run your development environment!

This is awesome for testing deployments while developing Cloudflare Workers, but also in pre-merge testing and integration testing.

Let me explain how to enable automation next. Check out the Automate tab.

Every test you create can be expressed with YAML. I know you love YAML, quit complaining! 😄

With this test definition you can trigger the same test via the CLI either locally or in any CI pipeline of you choice.

To try it locally, create a directory called test in the root directory.

Paste this into a file called test-api.prod.yaml.

type: Test
spec:
  id: WMGTfM2Sg
  name: Test API Prod
  trigger:
    type: http
    httpRequest:
      method: POST
      url: https://pokemon-api.<YOUR_URL>.workers.dev/api/pokemon?id=13
      headers:
      - key: Content-Type
        value: application/json
  specs:
  - selector: span[tracetest.span.type="faas" name="POST" faas.trigger="http"]
    name: Validate cold start
    assertions:
    - attr:faas.coldstart = "false"
  - selector: "span[tracetest.span.type=\"http\" name=\"GET: pokeapi.co\"]"
    name: Validate external API.
    assertions:
    - attr:http.response.status_code = 200
  - selector: "span[tracetest.span.type=\"general\" name=\"D1: Add Pokemon\"]"
    name: Validate Pokemon name.
    assertions:
    - attr:pokemon.name = "weedle"

Since you already have the Tracetest CLI installed, running it is as simple as one command. You can copy the command for your environment in the Automate tab in Tracetest.

tracetest configure --organization <YOUR_ORG> --environment <YOUR_ENV> && \
tracetest run test --file ./test/test-api.prod.yaml --required-gates test-specs --output pretty

[Output]
SUCCESS  Successfully configured Tracetest CLI
✘ Test API Prod (https://app.tracetest.io/organizations/<YOUR_ORG>/environments/<YOUR_ENV>/test/WMGTfM2Sg/run/1/test) - trace id: 59775e06cd96ee0a3973fa924fcf587a
    ✘ Validate cold start
        ✘ #2cff773d8ea49f9c
            ✘ attr:faas.coldstart = "false" (true) (https://app.tracetest.io/organizations/<YOUR_ORG>/environments/<YOUR_ENV>/test/WMGTfM2Sg/run/1/test?selectedAssertion=0&selectedSpan=2cff773d8ea49f9c)
    ✔ Validate external API.
        ✔ #d01b92c183b45433
            ✔ attr:http.response.status_code = 200 (200)
    ✔ Validate Pokemon name.
        ✔ #12443dd73de11a68
            ✔ attr:pokemon.name = "weedle" (weedle)

    ✘ Required gates
        ✘ test-specs

What’s cool is you can follow the link and open the particular test in Tracetest and view it once it’s saved in the cloud. Here’s a guide on using this pattern for integration testing Cloudflare Workers in the docs.

Beyond Deployment Testing

In conclusion, today you learned how to craft production-ready Cloudflare Workers. You now know how to develop, troubleshoot, and test Cloudflare Workers in staging and production. You started from a boilerplate, built an import flow, integrated OpenTelemetry for distributed tracing, and used trace-based testing for integration and deployment testing.

Want more? Jump over to the docs to learn about:

If you get stuck along the tutorial, feel free to check out the example app in the GitHub repo, here.

Stay tuned for the next part of this series coming soon:

Part 2: Learn how to configure production troubleshooting and testing in Cloudflare Workers by using observability tools like Grafana and Jaeger.

Would you like to learn more about Tracetest and what it brings to the table? Visit the Tracetest docs and try it out by downloading it today!

Also, please feel free to join our Slack community, give Tracetest a star on GitHub, or schedule a time to chat 1:1.

Forem: Adnan Rahić

Custom OpenTelemetry Collectors: Build, Run, and Manage at Scale

As a normal human I find this hard…

Building custom OpenTelemetry collectors is a real need

Why build a custom collector?

The OpenTelemetry Distribution Builder (ODB)

Step 1 — Create a GitHub repo

Step 2 — Write a manifest.yaml

Step 3 — Automate the build with GitHub Actions

Step 4 — Download and run your collector

Step 5 — Remotely manage your custom collector with OpAMP

1. Install the Bindplane CLI

2. Create an API Key & set a default profile

3. Create an Agent Type in Bindplane

5. Install your custom collector from the Bindplane UI

6. Configure and manage your custom collector from the Bindplane UI

Step 6 — Iterate with confidence

Why this works so well

Future goals

Final thoughts

How to Build Resilient Telemetry Pipelines with the OpenTelemetry Collector: High Availability and Gateway Architecture

How to define High Availability (HA) with the OpenTelemetry Collector?

Configure Agent-Gateway High Availability (HA) with the OpenTelemetry Collector

Nginx Load Balancer

Load Balancing Exporter

Building Resilience into Your Collector

Retry & Persistent Queue

Batching

Kubernetes-native load balancing with HorizontalPodAutoscaler

Final thoughts

Bindplane Launch Week 1: The Daily Releases Begin June 2 🪂

What to Expect

Monday: Telemetry Routing

Tuesday: Custom Collectors

Wednesday: Scale

Thursday: Easy Processing

Friday: A…Hi?

How to Join

End-to-End Observability with Grafana LGTM Stack

What is End-to-End Observability, And Why Do We Need It?

1. Setting Up Instrumentation in Your Application

1. Adding Metrics to Your Application

2. Adding Logging to Your Application

3. Adding Traces to Your Application

4. Containerize the Application

2. Setting up Prometheus for the Node.js Application

1. Create the Prometheus Configuration File

2. Run Prometheus in a Docker Container

3. Verifying Metrics Collection

3. Setting up Loki for the Node.js Application

1. Add Loki to Your Docker Compose Configuration

2. Verify Loki's is Running

4. Setting up Tempo for the Node.js Application

1. Create a Tempo Configuration File

2. Add Tempo to Your Docker Compose

5. Enabling Observability in Grafana

1. Adding a Data Source in Grafana

2. Setting up Grafana for Visualization

6. End-to-End Testing Using Trace-Based Testing with Tracetest

1. Sign up to Tracetest

2. Set up the Tracetest Agent

3 . Ingest the Traces from Tempo to Tracetest

4. Automate the Tests with Tracetest CLI

Conclusion

Frequently Asked Questions

Q. What are the stages of end-to-end testing?

Q. Is Prometheus a visualization tool?

Q. What is "trace" in "trace-based testing"?

Q. What is Grafana and Prometheus?

Trace-based Testing With OpenTelemetry: Using Tracetest with OpenTelemetry

What is Distributed Tracing?

Tracing vs Logging

Traditional Testing vs. Trace-based Testing

Introduction to OpenTelemetry

Instrumentation in OpenTelemetry

Using Tracetest for Trace-based Testing

How does Tracetest work?

Setting up Tracetest

Trace-based Testing in a Node.js application using Tracetest

Wrapping Up: Trace-Based Testing for Easier Validation

Step 2 — Write a `manifest.yaml`

2. Create an API Key & set a `default` profile