Forem: Peter Nguyen

Visualize Tailscale Network Flow Logs with New Relic

Peter Nguyen — Sun, 26 Oct 2025 18:06:35 +0000

Introduction

Did you know that Tailscale network flow logs can be visualized in New Relic? Take a look at my dashboard:

Technical Details

To get started, you'll need to have Tailscale Enterprise, as this is the only option that lets you stream network flow logs.

While this is not officially supported (yet), network flow logs from Tailscale can be streamed to New Relic. The only thing you need is the log endpoint https://log-api.newrelic.com/log/v1 and your New Relic Ingest license key (ending with NRAL).

At a first glace, most people wouldn't know it can be done, thanks to the Tailscale admin dashboard only showing a few vendors to stream to:

Since we only require the log endpoint and the license key, I tried using these two parametes with each of the vendors to see what actually gets the log data into New Relic.

At the time of writing this, there are 7 supported stream options and results from my testing:

❌AWS S3
✅Axiom
🟡Cribl
❌Datadog
❌Elasticsearch
🟡Panther
❌Splunk

❌ - Streaming Fails or additional parameters required
🟡 - Works, but not optimal for New Relic
✅ - Works, and optimal for New Relic

AWS S3

This option requires AWS credentials such as access key id and secret access key. So, this option won't work to stream data into New Relic. One option, however, is to stream this into AWS S3, then forward the S3 logs into New Relic (inefficient).

Axiom

Axiom was the only option that streamed the network flow logs in near "real-time", with attributes trimmed. Let me explain:

The data for the Network Flow Lows using the Axiom option is streamed almost in real-time, rather than being sent at 1-minute intervals (Cribl and Panther options).

Since network flow logs can get heavy, you'll want to optimize the data ingest for this. If you look at the screenshot, the attributes are not prefixed with event. compared to the other two options that work. As you can imagine, for high log volumes, having the extra event. prefix can add up to excessive data ingestion with little value.

Cribl

This option works. Both configuration and network flow logs are sent to New Relic without issue. The only drawback here is the additional event. prefix in the attributes, which can quickly add up to data ingest. Another thing to note is the Network Flow Logs are sent once a minute.

Datadog

This option doesn't send any logs to New Relic.

Elastic Search

This option has an additional Username option that is required. Using a random username won't send any logs to New Relic.

Panther

Splunk

This option doesn't send any logs to New Relic.

Tailscale Logs in New Relic

Once log streaming for both Configuration and Network Flow Logs are enabled, the logs look something like this:

Here is what this looks like on the Tailscale Admin console:

Here's what I did to create my Tailscale Dashboard.

Network Flow Log Visualization

Utilizing the Sankey Diagram visualization I created with Plotly in my previous post, I used the following NRQL query to generate widget:

WITH aparse(`physicalTraffic`, '[{"src":"*","dst":"*","txPkts":*,"txBytes":*,"rxPkts":*,"rxBytes":*}]')
  AS (src,dst,txPkts,txBytes,rxPkts,rxBytes)
SELECT count(*)
FROM Log
WHERE `physicalTraffic` IS NOT NULL
FACET src, dst
SINCE 72 HOURS AGO LIMIT MAX

This query parses the log data and generates a Sankey Diagram showing all of the physical traffic in Tailscale.

Network Flow Traffic

You can also add a line chart to visualize the transmit/received bytes:

WITH aparse(`physicalTraffic`, '[{"src":"*","dst":"*","txPkts":*,"txBytes":*,"rxPkts":*,"rxBytes":*}]')
  AS (src,dst,txPkts,txBytes,rxPkts,rxBytes)
SELECT
  average(numeric(txBytes)) AS 'Transmit Bytes',
  average(numeric(rxBytes)) AS 'Received Bytes'
FROM Log WHERE `physicalTraffic` IS NOT NULL
TIMESERIES 90 SECONDS
SINCE '12:45-0400' UNTIL '13:15-0400'
LIMIT MAX

Network Flow Raw Data

Since the data is streaming as-in to New Relic, we can use anchor parsing to make the data look clean in a table like this:

WITH aparse(`physicalTraffic`, '[{"src":"*","dst":"*","txPkts":*,"txBytes":*,"rxPkts":*,"rxBytes":*}]')
  AS (src,dst,txPkts,txBytes,rxPkts,rxBytes)
SELECT
src AS 'Source IP',
dst AS 'Destination IP',
  numeric(txPkts) AS 'txPkts',
  numeric(txBytes) AS 'txBytes',
  numeric(rxPkts) AS 'rxPkts',
  numeric(rxBytes) AS 'rxBytes'
FROM Log
WHERE `physicalTraffic` IS NOT NULL
AND rxBytes IS NOT NULL
SINCE 6 HOUR AGO LIMIT MAX

Configuration Log Visualization

Here's another useful visualization to see Configuration Changes at a quick glance:

SELECT count(*) AS 'Event Actions'
FROM Log WHERE `target.type` IS NOT NULL
FACET `action`
SINCE 6 HOURS AGO LIMIT MAX

And for the actual data in a tabular format:

SELECT
  `action`,
  `actor.displayName` AS 'Name',
  `target.name` AS 'Target'
FROM Log
WHERE `target.type` IS NOT NULL
SINCE 6 HOURS AGO LIMIT MAX

Remarks

Overall, streaming both the configuration logs and network flow logs from Tailscale to New Relic was easy. In my scenario, I used the Axiom option. It would be nice if Tailscale had a New Relic option, or at least an "Other" option to add a generic URL and generic header to accommodate other vendors.

Monitor R Applications with an OpenTelemetry Collector

Peter Nguyen — Thu, 26 Sep 2024 01:36:53 +0000

R - Plumber Monitoring

I didn't find any OpenTelemetry packages for R, so for this to work, I've made use of the openmetrics package that is essentially a Prometheus client for R using the OpenMetrics format.

Since we are dealing with Prometheus, I've also made use of the OpenTelemetry Contrib Collector, which has a Prometheus receiver built-in. Note, the base OpenTelemetry Collector does not have a Prometheus receiver built-in.

Prerequsites

\1. Download and install R.

\2. For Windows: Add this to a Path environment variable (change version as needed)

C:\Program Files\R\R-4.3.0\bin

\3. Launch R.exe or R.app and install the following packages:

install.packages("openmetrics")
install.packages("plumber")
install.packages("httr")

\4. Download otelcol-contrib and extract somewhere.

Plumber

Before starting, save these files, as we'll be using them as an example.
plumber.R

# plumber.R

#* Echo back the input
#* @param msg The message to echo
#* @get /echo
function(msg="") {
  list(msg = paste0("The message is: '", msg, "'"))
}

#* Plot a histogram
#* @serializer png
#* @get /plot
function() {
  rand <- rnorm(100)
  hist(rand)
}

#* Return the sum of two numbers
#* @param a The first number to add
#* @param b The second number to add
#* @post /sum
function(a, b) {
  as.numeric(a) + as.numeric(b)
}

#* Return the n-th Fibonacci number
#* @param n The n-th Fibonacci number
#* @get /fibonacci
fibonacci <- function(n="") {
  if (n == 0) {
    return(0)
  } else if (n == 1) {
    return(1)
  } else {
    return(fibonacci(as.numeric(n)-1) + fibonacci(as.numeric(n)-2))
  }
}

plumberSrv.R

library(openmetrics)
library(plumber)

register_default_metrics()
srv <- plumber::plumb("plumber.R")
srv <- register_plumber_metrics(srv)
srv$run(host="0.0.0.0", port=c(8000))

\1. Run this in a terminal to start the Plumber server. You may need to allow this app over the firewall.

RScript plumberSrv.R

\2. Go to http://127.0.0.1:8000/__docs__/ to generate some traffic.

\3. Go to http://localhost:8000/metrics to see the following metrics.

metric name	metric type
process_cpu_seconds_total	counter
process_cpu_seconds_total_created	gauge
http_request_total	counter
http_request_total_created	gauge
http_request_duration_seconds	histogram
http_request_duration_seconds_created	gauge

\4. Save this YML file as otel-config_r-plumber.yml. Since I'll be visualizing this data in New Relic, I've used an OTLP endpoint from New Relic, as well as an Ingest License Key to send these metrics to New Relic.

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'my-plumber-api'
          scrape_interval: 5s
          static_configs:
            - targets: ['localhost:8000']

processors:
  metricstransform/plumber:
    transforms:
      include: http_request_duration_seconds
      action: update
      new_name: http.server.duration
      operations:
        - action: experimental_scale_value
          experimental_scale: 1000
        - action: update_label
          label: path
          new_label: http.target

  attributes/plumber:
    actions:
      - key: service.instance.id
        action: insert
        value: localhost-osx
      - key: tags.team
        action: insert
        value: datacrunch

exporters:
  otlp:
    endpoint: https://otlp.nr-data.net:4317
    headers:
      api-key: ${NEW_RELIC_LICENSE_KEY}

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [metricstransform/plumber, attributes/plumber]
      exporters: [otlp]

\5. Run the OpenTelemetry Contrib Collector with the otel-config_r-plumber.yaml

./otelcol-contrib.exe --config=file:'./otel-config_r-plumber.yml'

New Relic Example

Go to APM & Services > OpenTelemetry to see the Plumber entity.

Check the Summary > Metrics page to see basic information

Click on Transactions on the left to see additional metrics.

Finally, go to the Metrics Explorer to see everything else.

Remarks

Using a similar methodology, I was able to repeat this for Shiny applications running with R, as well as batch jobs running in R. Check out these examples here:

Plotly Charts in New Relic

Peter Nguyen — Fri, 13 Sep 2024 15:09:47 +0000

Introduction

New Relic comes with several chart types you can use on a dashboard, line, stacked bar, pie, histogram, heatmap, just to name a few. However, I needed a "grouped bar chart" and there was no option for this. Luckily, New Relic has a programmability platform that lets you create custom visualizations.

Here's a link to my GitHub repository with the grouped bar charted created with Plotly: https://github.com/pnvnd/nr1-custom-visualizations

Example

This first example groups the data by application name, then country code as seen in the FACET clause.

This second example groups the data by country code, then application name:

General Instructions

You'll need git, npm, and nr1 to get started. Check to make sure everything is installed correctly.

git --version
npm -v
nr1 version

Next, clone the repository, install npm packages, and publish the nerdpack.

git clone https://github.com/pnvnd/nr1-custom-visualizations.git
cd nr1-custom-visualizations
npm install
nr1 nerdpack:uuid --generate --force
nr1 nerdpack:publish

In the New Relic console, go to Apps > Go to Apps and Visualizations

Search for the nerdpack, in this case Custom Visualizations.

Click Add this app

Select accounts that should have access to this Nerdpack and click Update on the top right.

Go back to Apps > Custom Visualizations

Select the Grouped Bar Chart to start using the Plotly chart

This particular grouped bar chart must have two FACET items. Here's an example:

Once you are satisfied with your query, you can add this to a dashboard.

Remarks

I've previously built a similar grouped bar chart with Recharts (simple bar chart) and here is the side-by-side comparison of what it looks like on a New Relic dashboard.

Plotly charts are interactive where you can zoom in and out, and save a screenshot of the chart.

Send incident.io Status to New Relic with Synthetics

Peter Nguyen — Fri, 06 Sep 2024 22:07:28 +0000

Here's a quick way to monitor a status page hosted by incident.io with New Relic. Specifically, I'll be using Synthetic monitoring to achieve this with a Scripted API test.

After digging a bit, it looks like data from a typical public status page hosted by incident.io is in this format: https://status.incident.io/proxy/statuspage-name

When you go to this page, you'll get something that looks like this:

Since this is structured JSON data, we can format this and ingest this data as an event in New Relic. Here's the full script

You can also find this here:
https://github.com/pnvnd/nodejs-synthetics/blob/main/scripted_api/incidentio.js

Or check it out here:


const got = require('got');
const statuspage = 'my-statuspage'

const account_id = $secure.ACCOUNT_ID
const ingest_key = $secure.INGEST_KEY

async function getStatusPage() {
  const url = `https://status.incident.io/proxy/${statuspage}`; 
  let resp = await got(url);

  if (resp.statusCode == 200) {
    let data = JSON.parse(resp.body);

    // Create a map for quick lookup of affected components by component_id
    let affectedComponentsMap = {};
    for (let affected of data.summary.affected_components) {
      affectedComponentsMap[affected.component_id] = affected.status;
    }

    // Update components with status if they are affected
    for (let component of data.summary.components) {
      component["eventType"] = "IncidentIOSample";
      component["statuspage"] = `${statuspage}`;
      component["status"] = affectedComponentsMap[component.id] || "operational";
    }

    got.post(`https://insights-collector.newrelic.com/v1/accounts/${account_id}/events`, {
        headers: {
          'Api-Key': `${ingest_key}`,
          'Content-Type': 'application/json'
        },
        json: data.summary.components
      });

      console.log(data.summary.components)

  } else {
    console.log(resp.body);
  }
};

getStatusPage();

Validate the script is working in New Relic then click Save.

Let the synthetic monitor run for a bit and check the data. Using NRQL, you can query your data with something like this:

SELECT * FROM IncidentIOSample

Another way of looking at this data is to view by component name:

SELECT latest(status)
FROM IncidentIOSample
WHERE statuspage='my-statuspage'
FACET name

Now that you have this data in New Relic, you can create alerts and stay on top of incidents hosted by public incident.io status pages.

Monitoring Jenkins pipelines using OpenTelemetry and New Relic

Peter Nguyen — Mon, 20 Mar 2023 19:01:00 +0000

Shift left by monitoring your continuous integration and continuous deployment pipelines for errors.

Jenkins is a popular open-source automation tool for implementing continuous integration (CI) and continuous development (CD) workflows called pipelines. Continuous integration and continuous deployment are development practices that can help you deploy your applications faster—and with fewer bugs. But if errors or other issues come up in your pipelines, it can negatively impact deployment and other steps in the DevOps lifecycle. That's why it’s important to monitor not just your applications but also the automation processes around development, including your CI and CD tools. This can help you shift left, which means incorporating more testing on the left side of the DevOps lifecycle, as shown in the next image.

Shifting left isn't just about testing your code and automating the build and deployment process. It's about monitoring and testing your CI and CD pipelines, too.

In this post, you’ll learn how to monitor your Jenkins jobs and pipelines using the Jenkins OpenTelemetry plugin and a New Relic quickstart integration. After you are finished, you'll be able to:

Visualize key metrics on your Jenkins pipelines, such as how long your jobs are taking and error rates.
Use distributed tracing to pinpoint where issues are coming from in your pipelines.
Create alerts on your Jenkins pipelines.

The next image shows a New Relic dashboard with some of the Jenkins metrics you’ll be able to visualize.

Install the Jenkins quickstart dashboard for a pre-built view of your key pipeline metrics.

With this integration, you can shift left and get observability for the build and deploy phases of the DevOps lifecycle. Check out the next video or continue reading to get started.

Prerequisites

If you don’t have Jenkins yet, you need to install it. Alternatively, you can use Docker to follow along with this example. If you’re using Docker, you can get an instance of Jenkins up and running with the following command:

docker run --name jenkinsci -d -e OTEL_LOGS_EXPORTER="otlp" -p 8080:8080 jenkins/jenkins:lts

Setting up the OpenTelemetry plugin

Log into Jenkins (usually http://localhost:8080) and log in with your username and password.
From the Jenkins dashboard, select Manage Jenkins from the lefthand pane.
Select Manage Plugins.
Select the Available tab and type OpenTelemetry in the search bar.
Select the OpenTelemetry checkbox and choose Install without restart.
After the installation is complete, select Restart Jenkins when installation is complete and no jobs are running.
Log into Jenkins again and select Manage Jenkins.
Under System Configuration, select Configure System.
Scroll down to the OpenTelemetry section and change the following settings:
- OTLP Endpoint should be https://otlp.nr-data.net:4317.
- Authentication should be Header Authentication.
- The Header Name should be api-key.
- For the Header Value, select Add > Jenkins. You’ll need to create a secret text with the following fields:
  - For Kind, choose Secret text.
  - The Secret should be your New Relic license ingest key.
  - ID can be left blank.
  - Add a description such as New Relic INGEST - LICENSE key.
  - Select Add.
After you’ve added your credentials, you’ll be able to select New Relic INGEST - LICENSE Key from the Header Value dropdown menu.
Select Save.

Sending Jenkins data to New Relic

After you’ve set up your New Relic credentials in Jenkins, you’re ready to send data to New Relic. Let’s start by setting up a test pipeline that will automatically fail. By testing a pipeline, you’re making sure that the connection to New Relic is set up properly. You can set up a test pipeline with the following steps.

In the Jenkins dashboard, select New Item.
Enter an item name (the name in this example is test) and select Pipeline. Then select OK.
Scroll to the bottom, select Scripted Pipeline, and then select Save.

Some older versions of Jenkins don’t have sample pipelines that you can choose from the list. If this is the case for you, paste in the following script and select Save.

pipeline {
  agent any
  stages {
    stage('Build') {
      steps {
        git 'https://github.com/jglick/simple-maven-project-with-tests.git'
        sh "mvn -Dmaven.test.failure.ignore=true clean package"
      }
      post {
        success {
          echo "success"
        }
      }
    }
  }
}

Select Build Now. This sample pipeline should fail, which is the expected behavior.
Log into New Relic, go to Services - OpenTelemetry, and select jenkins.

Now you can go to Transactions to see your data. You should see all of the pipeline build steps, including both successes and failures. The next image shows a Jenkins dashboard in New Relic. It includes information on top transactions, top 20 transactions, and throughput.

You can also go to Distributed Traces to see data from your Jenkins pipeline. In this example, you can see exactly where the pipeline starts failing, which is helpful for creating alerts. The next image shows a New Relic dashboard with specific spans in a Jenkins pipeline distributed trace.

If you’ve set the environment variable to OTEL_LOGS_EXPORTER="otlp", you can see your Jenkins console logs in the Logs tab. The next image shows logs related to a Jenkins pipeline.

Visualize your Jenkins data

Once you have Jenkins data flowing into New Relic, you can add our quickstart dashboard to get an overview of your Jenkins environment. For example, you can see a summary of the number of Jenkins instances available, the number of successful jobs, the top errors, and the job queue status. To get this dashboard, simply follow these steps:

Go to the Jenkins quickstart in New Relic Instant Observability, and select Install now.
Select an account and select Begin installation.
If you've already completed the earlier steps in this tutorial, select Done to move on to the next step.
The quickstart deploys the resources to your account. Select See your data to get to the dashboard.

Install the Jenkins quickstart dashboard for a pre-built view of your key pipeline metrics.

After you're done, you'll have complete visibility into your Jenkins pipelines, from high-level metrics to logs and distributed traces. You can also optionally choose to add alerts when certain thresholds are met, such as when jobs are taking too long or when the error rate gets too high. To do so, you can follow along with the docs to first create a policy and then set conditions on that policy. You can send alerts to the notification channels of your choice. Options include Slack, PagerDuty, email, and more.

Sending your Jenkins pipeline data to New Relic gives you observability into the building and deployment phases of the DevOps lifecycle, giving you more insights into your DevOps practices and helping you shift left.

You can read the full blog post at New Relic here.