Forem: Harry Kimpel

Build AI Agents You Can Actually Trust — Hackathon in Mountain View 🚀

Harry Kimpel — Wed, 11 Mar 2026 20:12:07 +0000

You Founded a Startup. Your AI Agents Are Hallucinating. Your Investors Are Watching

Sound fun? It is, actually.

On March 27, we're hosting a free, in-person hackathon at the Microsoft Mountain View Campus (1045 La Avenida St, Mountain View, CA) where you'll build an AI-powered travel planning assistant from scratch — and then make it production-ready with real observability and security controls.

This is part of Microsoft's What The Hack series: collaborative, challenge-based hackathons where you learn by doing, not by watching someone else's screen.

🌍 The Scenario: Welcome to WanderAI

You've just founded WanderAI, a travel planning startup. Your customers describe their dream trip, and your AI agents craft personalized itineraries.

But here's the catch — your investors want answers:

🔍 Are the agents making good recommendations?
⚡ How fast are they responding?
🚨 When something breaks, can we debug it?
✅ Are the travel plans actually... good?

Your mission: go from "cool demo" to "production-ready AI service" in a single day.

🛠️ What You'll Build (and Learn)

The hack is structured as 8 progressive challenges:

#	Challenge	What You'll Do
00	Prerequisites	Set up your GitHub Codespace
01	Master the Foundations	Understand agent architecture, tools & orchestration
02	Build Your MVP	Create a Flask web app + your first AI agent with tool calling
03	Add OpenTelemetry	Instrument with built-in telemetry, verify traces in console & New Relic
04	New Relic Integration	Custom spans, metrics, and correlated logging
05	Monitoring Best Practices	Dashboards, alerting, and production monitoring patterns
06	LLM Quality Gates	Build evaluation tests and CI/CD quality gates for AI outputs
07	Platform Security	Configure Microsoft Foundry Guardrails
08	App-Level Security	Prompt injection detection and blocking

By the end, you'll have a fully instrumented, observable, and secure multi-agent AI system. Not bad for one day.

🧰 The Tech Stack

Microsoft Agent Framework — for building multi-agent orchestrations
OpenTelemetry — the open standard for traces, metrics, and logs
New Relic — for sending, visualizing, and alerting on all that telemetry
Azure — the cloud backbone
Python + Flask — the app layer
GitHub Codespaces — zero local setup headaches

🍕 The Details

When: March 27, 2026
Where: Microsoft Mountain View Campus, 1045 La Avenida St, Mountain View, CA
Cost: Free
Food: Provided
Duration: ~3–5 hours
What to bring: Your laptop and curiosity

Who Is This For?

Anyone who's building (or thinking about building) AI agents and wants to understand what's happening under the hood. Whether you're a backend engineer, an ML practitioner, a platform engineer, or just AI-curious — this hack meets you where you are.

No prior experience with New Relic or OpenTelemetry is required. Basic Python and web dev knowledge is helpful.

🎟️ Register Now

Space is limited — grab your seat before it fills up:

👉 Register here

See you in Mountain View! 🏔️

Unleashing the Power of Monitoring: Master Your WordPress with New Relic

Harry Kimpel — Tue, 25 Nov 2025 19:58:22 +0000

WordPress powers countless websites across various domains, offering incredible versatility. This Content Management System (CMS) is the undisputed leader in the CMS market, powering an impressive 43.6% of all websites globally, according to these statistics. With over 810 million websites built on the platform and hundreds more launching daily (500+), its adoption continues to surge. This widespread use gives WordPress a massive 62% CMS market share, significantly outpacing its rivals.

However, even the most robust WordPress sites can face performance challenges. Slowdowns are often caused by factors such as slow-loading plugins, database connection issues, infrastructure capacity problems, network trouble, large page assets (like images or fonts), and broken links. This is why robust monitoring is essential for maintaining a fast, reliable, and user-friendly website.

The question now is: what should be monitored in WordPress?

To truly master your WordPress environment, monitoring needs to extend across the entire stack:

WordPress Application Code: The core of your site and any custom code or plugins you've added.
Underlying Infrastructure: This includes the hosts, servers, or virtual machines (VMs) that power your site.
(Apache) Web Server: Monitoring the web server ensures it's handling requests efficiently.
(MySQL) Database: Database performance is critical, as slow queries can quickly bottleneck your site.
End-User Experience (Frontend): How your website performs for your actual users, measuring factors like page load times.
Business Objectives: Tracking metrics like visitor count, device usage, and post performance, which tie directly to your business goals.

How to Monitor WordPress

You have several powerful options for implementing comprehensive WordPress monitoring:

1. New Relic Observability Platform

New Relic offers a powerful full-stack WordPress integration that monitors your application's performance. It helps you diagnose issues and optimize your code by leveraging New Relic's existing PHP, Apache, and MySQL integrations. This provides pre-built dashboards with crucial metrics like transactions, visitors, and call duration. The installation process is often surprisingly quick, with agents installed in minutes.

2. OpenTelemetry (Open-Source Standard)

OpenTelemetry (CNCF project) is a vendor-agnostic set of APIs, libraries, and a collector service designed to standardize how you collect telemetry data (Traces, Metrics, and Logs) from your applications.

Traces: Show the path of a request through your application and services.
Metrics: A measurement captured at runtime, such as Avg. CPU, Throughput, or Max. Memory.
Logs: A recording of an event.

The OpenTelemetry WordPress extension monitors performance and, unlike some WordPress setups, relies on a composer for managing dependencies, including the OpenTelemetry SDK and an exporter to send telemetry data to an observability platform of your choice.What Monitoring Reveals

Benefits of WordPress monitoring

Implementing monitoring gives you deep visibility into your site's performance:

Frontend Monitoring

This focuses on the user experience, revealing how quickly pages load and identifying performance bottlenecks visible to the end-user.

New Relic monitoring capabilities automatically include Google’s lighthouse metrics.

This allows you to either visualize a comparison between frontend and backend duration or dive deeper into the breakdowns of the frontend sections of your page view load time.

Another key aspect of frontend aka real user monitoring is New Relic’s Geography UI page. It provides a world view with color-coded performance information about your frontend experience in cities, regions, and countries anywhere around the world. The map shows your user data by region, so you can visualize your traffic and error hotspots alongside device type information. You can also examine critical Core Web Vitals data so that you can prioritize areas around the globe needing attention.

Backend Analysis

Monitoring provides detailed insight into your server-side performance, allowing you to examine:

Golden Signals: Web transaction time, throughput, errors

Especially when seeing spikes, a timeseries view provides visual representations of patterns and anomalies.

Transactions: The processing time for various key requests

Database: Slow or inefficient database queries

Hooks: Performance of WordPress action and filter hooks

Traces: Detailed views of a single request's journey through your system

Business Metrics

Beyond technical performance, monitoring allows you to track metrics that directly impact your business:

Visitors: Analyze user traffic and trends

Devices: See which devices your visitors are using

Posts: Monitor the performance and traffic of specific content

Backend Users: Track activity and performance related to logged-in users and administrators.

Conclusion: Why, What, and How

Effective WordPress monitoring answers three key questions:

Why Monitor? To resolve slowdowns, address infrastructure issues, and ensure a positive end-user experience.
What to Monitor? The technical aspects (code, infrastructure, database), the end-user experience, and business objectives.
How to Monitor? By utilizing powerful options like the New Relic observability platform or the vendor-agnostic OpenTelemetry standard.

Ultimately, monitoring is about understanding where your application is excelling and where it needs attention, allowing you to proactively optimize performance and achieve your business goals.

Ready to gain deep insights into your WordPress performance?

Get started with New Relic today and leverage the power of comprehensive observability, whether through our native integrations or the flexibility of OpenTelemetry.

Optimizing Kafka Tracing with OpenTelemetry: Boost Visibility & Performance

Harry Kimpel — Tue, 25 Nov 2025 19:55:31 +0000

Ideally, you should be using distributed tracing to trace requests through your system, but Kafka decouples producers and consumers, which means there are no direct transactions to trace between them. Kafka also uses asynchronous processes, which have implicit, not explicit, dependencies. That makes it challenging to understand how your microservices are working together.

However, it is possible to monitor your Kafka clusters with distributed tracing and OpenTelemetry. You can then analyze and visualize your traces in an open-source distributed tracing tool like Jaeger or a full observability platform like New Relic. In this post, I will leverage a simple application to show how you can achieve this.

Design Considerations & Guidelines

OpenTelemetry typically comes in two flavors:

When I talk about these flavors, I typically use the analogy above. You can either buy a ready-made cake and enjoy it or buy all the ingredients and make the cake yourself. With OpenTelemetry, the approach is very similar and the flavors are:

Zero-code instrumentation: in this approach, you will use an OpenTelemetry agent and attach it to your application at startup time. This agent will then do its magic and automatically (without any source code changes) provide a lot of telemetry signals (metrics, traces and logs) and insights into your application.
- Pros:
- Getting started quickly
- No source code changes
- Cons:
- Limited customization
- Depth of visibility into your application may be limited
Manual instrumentation: this option requires you to add some dependencies and packages to your source code that you need to manage as part of your regular software development lifecycle (SDLC). However, this also allows you to be more specific and custom about your instrumentation. You can easily add custom metrics, traces, attributes to your telemetry.
- Pros:
- Way more flexible with customizing telemetry
- Easily able to add, remove and tweak the depth of your instrumentation
- Cons:
- Dependencies in your source code
- More effort to implement

Sample application

The sample application (available in this public GitHub repository) that I am using in this blog is based on this high-level architecture:

It contains these components:

kafka-java-producer: a Java Spring Boot application that produces messages into a Kafka topic
Kafka broker
kafka-java-consumer: a Java Spring Boot application that subscribes to a Kafka topic and reads messages from it. This component also makes calls to an external REST API service (that is not in our control)
kafka-java-service: a downstream Java Spring Boot application that is being called from the consumer service

Zero-code instrumentation

Let’s start with zero-code instrumentation, aka automatic instrumentation.

Configuration

Each of the different services contain a run.sh script to get the service up and running. The script looks like this:

The key line in this is the first one. Here we are defining the JAVA_TOOL_OPTIONS and configuring the -javaagent to point to the location of the OpenTelemetry Java agent.

The next three lines configure how we want to deal with the different telemetry signals. In our case, I define the traces, metrics and logs to be exported via OpenTelemetry Line Protocol (OTLP).

There are three additional environment variables that are quite important to configure:

OTEL_EXPORTER_OTLP_ENDPOINT: the target system where we want to export the data to, i.e. our telemetry backend. In my case, this is for sure New Relic and so I configure New Relic’s native OTLP endpoint
OTEL_EXPORTER_OTLP_HEADERS: the above exporter endpoint is an open API, so we need to configure an API key. In the case of New Relic, this is a New Relic license key.
OTEL_SERVICE_NAME: ideally, we want to give the service a meaningful name, so that New Relic can create an appropriate entity from it.

This is basically all we need to configure. Everything else is dealt by the OpenTelemetry Java agent. No need to change anything in our source code.

Observability

Let’s see what level of visibility into the services we can achieve from zero-code instrumentation.

When navigating to my New Relic account, I can see all services reporting into separate entities.

Let’s start by exploring the kafka-java-producer service.

The Summary view offers a great overview of all the most important telemetry and metrics I should be focusing on.

As part of this blog, I am mostly interested in the Distributed Tracing section, so let’s dive deeper into this area.

By looking at a single trace, this allows me to view the detailed information on how long this specific trace took to execute and where the time was spent.

We also automatically draw an Entity map of all the different services involved in a given trace.

The interesting area that I want to draw your attention to lies in the trace and span breakdown. You can see how the trace gets initiated on the producer, the consumer then picks up the message and how the consumer then also makes two separate calls to the downstream service.

What is interesting here is the span that says “Uninstrumented time”. This is code in the consumer where the agent was not able to capture some more detailed information about what is going on in its internal methods.

This already shows the limits of zero-code instrumentation. The agent by default will not instrument all the various methods and source code, but rather stops - by design - at some level to get deeper visibility into your code.

Manual instrumentation

In the previous section, you saw how zero-code instrumentation has some limits when it comes to visibility into your application. This is exactly where manual instrumentation comes into play.

Configuration

I have configured the same application, but this time, no agent at all is configured when starting the application.

I simply use the Maven wrapper to run the application.

The other configuration details are then part of my application.properties:

These properties are then used in my Spring Boot application code to define the configuration for OpenTelemetry for traces, metrics and logs.

Observability

Before I jump into the details of how I implemented some manual instrumentation, let’s have a look at the result first.

Do you notice how the span, which previously was called out with “Uninstrumented time”, now shows much more detailed information? I now can see these additional spans:

ExecuteLongRunningTask
WhyTheHeckDoWeSleepHere
SomeTinyTask
AnotherShortRunningTask

The one that says “WhyTheHeckDoWeSleepHere” seems to be taking the most time. No wonder, as the name suggests 😉.

Let’s have a look at the source code to reveal the manual instrumentation I put in place.

In the method named ExecuteLongRunningTask I have created a new span on the current tracer by using the spanBuilder() Method.

In addition to that, you may also notice that - just for the fun of it - I created another span called “WhyTheHeckDoWeSleepHere” that contains an artificial unit of work or rather a sleep instruction on the current thread.

These concepts to leverage the OpenTelemetry SDK allow me to be much more specific in getting insights and details into my application and source code. But, as you can imagine, also have the caveat that I need to have some dependencies and custom code available in my source code.

Conclusion

I hope I was able to show you how easy it can be to leverage OpenTelemetry in order to get insights into your application and services. We looked into zero-code instrumentation to get started without any code changes, but the level of details may be limited. We then also looked into manual instrumentation. This allowed us to be more specific and customize the instrumentation, but the effort to get started is a little higher.

I encourage you to have a look into OpenTelemetry and its fascinating capabilities. Let me know your thoughts and please get in touch if you have any questions or need further information.

Happy coding!

Why I'm Excited for .NET Conf 2024

Harry Kimpel — Tue, 12 Nov 2024 11:45:46 +0000

Greetings fellow .NET enthusiasts and tech aficionados! With the highly-anticipated .NET Conf 2024 just around the corner, I find myself reflecting on how far we've come in the world of .NET development and what thrilling innovations are yet to come.

A Journey Through .NET

Let's take a quick stroll down memory lane. My experience with .NET technologies dates back to 2002, when I first dipped my toes into the vast sea of possibilities offered by .NET. Over the years, I've seen it transform from a promising framework into a powerhouse that underpins countless applications worldwide. Every upgrade has brought something new and exciting, fueling my passion for staying on the cutting edge of technology.

Spotlight on .NET 9

Fast forward to today, one of the highlights I'm most eager about at this year's conference is the release of .NET 9. The advancements and features in .NET 9 promise to revolutionize how we build, deploy, and manage applications. From performance improvements to enhanced security features, .NET 9 is set to redefine efficiency, enabling developers to create more robust and scalable solutions.

But what truly excites me is the introduction of and this year's updates on .NET Aspire, a suite of tools designed to streamline the development process, especially for large-scale applications. This groundbreaking addition not only underscores Microsoft's commitment to innovation but also provides developers with the power to push boundaries and elevate their craft.

Exploring Blazor and .NET MAUI

Another reason I'm counting the days to .NET Conf 2024 is to learn more about the latest advancements in Blazor and .NET MAUI. Both technologies have been game-changers in their own right, and I'm particularly interested in seeing how new features will enhance our ability to create cross-platform applications with seamless user experiences.

The Role of AI and Emerging Technologies

This year, there's a dedicated spotlight on AI and its integration into the .NET ecosystem. Artificial Intelligence is a frontier that holds immense potential, and exploring its convergence with .NET will undoubtedly open doors to innovative applications and smarter solutions. The prospect of using .NET to enhance AI capabilities and vice versa is an exhilarating thought, aligning perfectly with the forward-thinking spirit that drives our community.

Connecting With the .NET Community

Beyond the technical sessions and keynotes, .NET Conf offers an unparalleled opportunity to connect with fellow developers and technology leaders. Engaging with this vibrant community is always a highlight of the conference for me—it's where shared knowledge, collaborations, and lifelong friendships are born.

Final Thoughts

.NET Conf 2024 promises to be a landmark event, brimming with insights, revelations, and opportunities to explore the future of .NET development. I hope you're as excited as I am to discover what's in store and to continue pushing the boundaries of what's possible with .NET technologies.

If you’re attending the conference, I’d love to hear what sessions you’re looking forward to and how .NET continues to inspire your work. Let's connect and share our experiences as we chart our course through the evolving landscape of .NET development. Here's to exploring new horizons with .NET!

Feel free to reach out or drop your thoughts in the comments below. Until then, happy coding!

Observability as code for AI apps with New Relic and Pulumi

Harry Kimpel — Mon, 14 Oct 2024 18:30:45 +0000

To read this full article, click here.

AI applications are complex and distributed, making effective monitoring challenging. Combining the New Relic intelligent observability platform with Pulumi's infrastructure-as-code and secret management solutions allows for an end-to-end "observability as code" approach. This method enables teams to:

Define artificial intelligence (AI) and large language model (LLM) monitoring instrumentation along with cloud resources programmatically.
Securely manage API keys and cloud account credentials.
Automatically deploy New Relic instrumentation alongside AI applications and infrastructure.

Benefits include:

Consistent monitoring across environments
Version-controlled observability configuration
Easier detection of performance issues
Deeper insights into AI model behavior and resource usage

The observability-as-code approach helps developers maintain visibility into their AI applications as they scale and evolve.

What is Pulumi?

Pulumi provides a range of products and services for platform engineers and developers, including:

Pulumi infrastructure as code (IaC): An open-source tool for defining cloud infrastructure. It supports multiple programming languages. For example, you can use Python to declare AWS Fargate services, Pinecone indexes, and custom New Relic dashboards.
Pulumi Cloud: A hosted service that provides additional features on top of the open-source tool, such as state and secrets management, team collaboration, policy enforcement, and an AI-powered chat assistant, Pulumi Copilot.
Pulumi Environments, Secrets, and Configuration (ESC): This ensures the secure management of sensitive information necessary for observability. For example, you can manage New Relic, OpenAI, and Pinecone API keys and configure OpenID Connect (OIDC) in AWS. This service is also part of Pulumi Cloud.

Pulumi allows teams to version control their observability configurations and infrastructure definitions. This ensures consistency across environments and simplifies the correlation of application changes with monitoring updates and underlying infrastructure modifications.

How to achieve observability as code with New Relic and Pulumi

In this guide, you’ll add AI and LLM monitoring capabilities to an existing chat application by configuring the New Relic application performance monitoring (APM) agent, and by defining New Relic dashboards in Python using Pulumi.

The final version of the application and infrastructure referenced throughout the guide resides in the AI chat app public GitHub repository.

Before you start

Ensure you have the following:

A New Relic account and a valid New Relic license key
A Pulumi Cloud account
The Pulumi CLI installed locally

All the services used throughout this guide qualify under the respective free tiers.

Explore the OpenAI demo application

The OpenAI demo application used throughout this guide is written in Node.js for the backend and Python for the frontend. It interacts with OpenAI to generate various gameplays through generative conversational AI interactions. It uses the public OpenAI platform to call its API to access different LLMs like GPT-3.5 Turbo, GPT-4 Turbo, and GPT-4o.

Below is a screenshot of a "higher or lower" gameplay:

The demo simulates a retrieval-augmented generation (RAG) flow, which commonly looks up information that the AI either doesn't know or might hallucinate. The app stores a handful of common game names and instructions in a Pinecone vector database and then uses it as an embedding when calling OpenAI.

This application is configured to observe its performance, such as traces, metrics, and logs. It leverages New Relic's latest innovation in monitoring AI interactions, such as requests and responses. This capability ensures compliance, promotes quality, and observes your AI costs.

Run the app locally with Docker Compose

The easiest way to run the chat on your local machine is via Docker Compose. Inspect the docker-compose.yml file in the ./app folder and create the required .env file as shown. Then, in your terminal, run:

cd app 
docker compose up \-d –build

By default, the web service will run on http://localhost:8888/.

Try out the API endpoints

The API backend provides various endpoints that expose all of its AI functionality. The frontend leverages these endpoints to simulate a flow of activity for end users to select a game and initiate a game interaction, that is, play the game, with OpenAI.

Follow these steps to simulate a higher or lower gameplay:

Open the web service http://localhost:8888/.
Click the Get Games button. A list of games is displayed.
In the Input field, copy higher or lower.
To retrieve a game prompt for the next step, click Get Game Prompt.
To send the OpenAI request to interact with the game, click Submit Prompt.
Enter your first guess into the Insert Your Game Interaction textbox. You’ll get a message about whether your guess was correct when compared to the number the AI picked. Repeat this step until you have guessed the correct number.

Configure New Relic agents with AI

The chat application is configured to capture telemetry using the New Relic APM agent for API and web services. This agent also leverages New Relic's latest innovation in monitoring AI interactions, such as requests and responses. This capability ensures compliance, promotes quality, and observes AI costs.

Enable the New Relic AI monitoring capabilities through the New Relic APM agent configuration or via environment variables:

NEW_RELIC_AI_MONITORING_ENABLED = TRUE
NEW_RELIC_SPAN_EVENTS_MAX_SAMPLES_STORED = 10000
NEW_RELIC_CUSTOM_INSIGHTS_EVENTS_MAX_SAMPLES_STORED = 100000

Before exploring the data streamed by the New Relic agents, you’ll use Pulumi to deploy the application to AWS and create custom dashboards. This will enable you to simulate global traffic interacting with the application, thus generating representative AI metrics to forecast performance and cost.

Manage your secrets with Pulumi ESC

While running everything locally is always a good first step in the software development lifecycle, moving to a cloud test environment and beyond involves ensuring .env files are securely available and all the application infrastructure dependencies are deployed and configured. The "Ops" side of DevOps comes into the picture, but it need not be a daunting task. You’ll leverage Pulumi ESC to manage the chat application's secrets.

Create a Pulumi ESC Environment

You'll create a Pulumi ESC environment to store all your .env secrets in Pulumi Cloud. This enables teams to share sensitive information with authorized accounts. Ensure that your New Relic license key, OpenAI token, and Pinecone API keys are handy.

In your terminal, run:

pulumi login
E=my-cool-chat-app-env
pulumi env init $E --non-interactive
pulumi env set  $E environmentVariables.NEW_RELIC_LICENSE_KEY 123ABC --secret
pulumi env set  $E environmentVariables.OPENAI_API_KEY 123ABC --secret
pulumi env set  $E environmentVariables.PINECONE_API_KEY 123ABC --secret

Load your .env file with the Pulumi CLI

Now that you’ve defined your ESC environment, you can consume it in several ways. For instance, you can populate our .env files by opening the environment using dotenv formatting:

cd ./app
pulumi env open $E --format dotenv > ./.env
docker compose up -d –build

The Pulumi commands may be scripted to run inside an Amazon EC2 instance or AWS Fargate. Next, you’ll define AWS resources using Pulumi IaC and Python so that you can run the application in AWS. To learn more, visit the Pulumi ESC documentation.

Generate infrastructure code with Pulumi Copilot

Nowadays, you don’t need to start from scratch when using infrastructure as code. Instead, you’ll use the power of generative AI to write Python code to declare all the cloud resources needed for our chat application. This entails declaring New Relic, AWS, and Pinecone resources.

Pulumi Copilot is a conversational chat interface integrated into Pulumi Cloud. It can assist with Pulumi IaC authoring and deployment. Let's have an intelligent conversation with Pulumi Copilot to help us start writing a Python-based Pulumi program that will also have access to the previously created ESC environment.

Prompt Pulumi Copilot with:

Can you help me create a new and empty Python Pulumi project called "my-cool-chat-app" with a new stack called dev? Add "my-cool-chat-app-provider-creds" to the imports in the dev stack

Declare New Relic AI LLM monitoring dashboards

A standard method for sharing dashboards is through the JSON copy and import feature. However, importing these through the console can result in reproducibility issues over time. For example, what happens when the dashboard JSON definition changes in an undesirable way? It becomes a challenge because there are no versioning details readily available.

Instead, using a declarative approach to "import" the JSON file unlocks several benefits. It allows for managing the dashboard's lifecycle (creation, deletion, updates) through code, thereby tracking changes over time. Also, it makes it easy to incorporate into deployment pipelines and share these across teams.

You have an AI/LLM monitoring dashboard JSON file and want to use it to create a new dashboard under our New Relic account. Let's continue our chat with Pulumi Copilot and prompt it to update the current solution:

Okay great! Can you update the Python code to deploy a New Relic dashboard based on an existing JSON file I will provide as input?

Declare a Pinecone index

A Pinecone index was needed beforehand to test the chat application locally. Let's ensure its existence before deploying the chat application to the cloud.

Let's ask Pulumi Copilot to define this resource on our behalf:

Thank you! I also need a serverless Pinecone index named "games" in the "default" namespace on AWS us-east-1 with 1536 dimensions. Can you generate the Python code to define this resource?

Declare an Amazon EC2 instance

To test the chat application in a cloud environment, you'll use an EC2 instance. Let's ask Pulumi Copilot to define all the AWS resources:

Perfect! I also want to deploy my chat application to an Amazon EC2 Linux instance. 
Create a VPC, security group, public subnet, and route table for the EC2 instance.
Ensure the security group allows inbound SSH traffic on port 22 from anywhere.
Ensure the EC2 instance is publicly accessible and runs a Docker Compose command to start the application.
Associate the EC2 instance with a public IP
Update the EC2 instance with a depends_on resource option for the route table association.

Deploy the application with Pulumi

You’ve asked Copilot to help with the Python infrastructure code, so it's time to deploy our application. In the chat window, expand the Pulumi Code drop-down. Click the Deploy with Pulumi button to create a new project.

Once you download the project, you'll add credentials, modify the code slightly, and deploy our application on AWS.

Click the Deploy with Pulumi button to create a new project. Then,
Choose the CLI Deployment deployment method and click Create Project.
Follow step 3 from the "Get started" steps in the next window to download the project onto your development environment.

Given that you already have a relatively small application in a GitHub repository, you’ll add a new empty folder at the root level named infra, where you'll unzip the contents of the Pulumi project. Compare your solution with the final version hosted on GitHub and fix any minor details Copilot may have overlooked.

Also, include the following changes from the final version shared:

Update the custom user_data script.
Update the dashboard.json to include your New Relic account id.
Optionally, include the Docker build code to build and push the images to Dockerhub.

To learn more, visit the Pulumi Copilot documentation.

In order for Pulumi to deploy all the declared resources, it needs access to your cloud accounts. These credentials will reside in a Pulumi ESC Environment named "my-cool-chat-app-provider-creds". Refer to the README to configure the Environment and set up the Python virtual environment before deploying everything via:

pulumi up --stack dev  --yes

It takes about a minute for all resources to be created. Once created, access the public URL displayed and run load tests.

Example partial output from the above command:

     Type                                Name                        Status              
 +   pulumi:pulumi:Stack                 my-cool-chat-app-dev-dev    created (57s)       
 +   ├─ newrelic:index:OneDashboardJson  my_cool_dashboard           created (2s)        
 +   ├─ pinecone:index:PineconeIndex     my_cool_index               created (7s)        
 +   ├─ docker-build:index:Image         ai-chat-demo-api            created (2s)        
 +   ├─ docker-build:index:Image         ai-chat-demo-web            created (3s)        
 +   ├─ aws:ec2:Vpc                      my_cool_vpc                 created (13s)       
 +   ├─ aws:ec2:Subnet                   my_cool_subnet              created (11s)       
 +   ├─ aws:ec2:SecurityGroup            my_cool_security_group      created (4s)        
 +   ├─ aws:ec2:InternetGateway          my_cool_igw                 created (1s)        
 +   ├─ aws:ec2:RouteTable               my_cool_route_table         created (2s)        
 +   ├─ aws:ec2:RouteTableAssociation    my-route-table-association  created (0.89s)     
 +   └─ aws:ec2:Instance                 my_cool_instance            created (24s)       

Outputs:
    url: "52.41.60.240"

Resources:
    + 12 created

Duration: 1m0s

Explore your New Relic AI LLM dashboards

To recap, the chat application is configured to observe its performance, such as traces, metrics, and logs, using the New Relic APM agent and the infrastructure agent. Now that we’ve simulated traffic, let's review the collected telemetry data in New Relic.

AI Response metrics

The AI Responses section will highlight key metrics when observing your AI/LLM applications. It includes data such as Total responses, Response time, Token usage per response, and Errors within your AI interactions. Time series graphs show you the same information with some more historical context.

The bottom part of that screen provides more insights into the requests and responses your end customers used to interact with the chat application.

AI Model comparison

Another very important aspect of New Relic AI monitoring is the Model Inventory section.

This view provides an intuitive overview of all your AI/LLM models being leveraged in our chat application. As you can see, we ran the same application with OpenAI models GPT-3.5 Turbo, GPT-4 Turbo, and GPT-4o. The AI model used is reflected in the chatModel variable for the API AI backend. This view shows you all the critical aspects of the performance, quality of responses, errors, and cost at a glance.

When combining the raw data from AI monitoring and custom lookup tables, New Relic also allows you to build custom dashboards (see Deploy the application with Pulumi for a reference to import custom dashboards). These custom dashboards can bring additional data, such as the actual cost (in $) for my OpenAI platform, and leverage the input and output tokens from our monitoring to calculate the total AI cost for running the chat application.

OpenAI custom dashboards

Next steps

In this guide, you explored an AI demo application powered by OpenAI and Pinecone. The application uses New Relic AI dashboards to monitor costs and other performance metrics. You used Pulimi Copilot to generate all the cloud components needed to run the chat application in AWS successfully while storing all sensitive information in Pulumi ESC.

Join us in the Observability as Code for AI Apps with New Relic and Pulumi virtual workshop, where you'll see the chat application, Pulumi, and New Relic in action.

Using .NET Aspire eShop application to collect all the telemetry

Harry Kimpel — Mon, 10 Jun 2024 08:15:37 +0000

To read this full article, click here.

Learn how to collect all the telemetry from the .NET Aspire eShop application and send it to an OpenTelemetry backend such as New Relic

.NET Aspire is the new kid on the block when it comes to an opinionated, cloud-ready stack for building observable, production-ready, distributed applications. Having a built-in dashboard for the monitoring data is nice during development. But how do you configure OpenTelemetry correctly to send it to an observability backend? This is what this blog post is all about. And you’ll also learn how to send custom attributes by leveraging OpenTelemetry SDKs.

.NET Aspire

.NET Aspire was first announced and introduced at .NET Conf 2023 Keynote. The challenges it tries to solve are:

Complex: Cloud computing is fundamentally hard.
Getting started: For new developers in this space, a first step into cloud native can be overwhelming.
Choices: Developers need to make a lot of choices.
Paved path: .NET did not have a golden paved path available for developers to build cloud-native applications.

This is exactly where .NET Aspire comes into play. It includes the following features as part of the stack:

Components: Curated suite of NuGet packages specifically selected to facilitate the integration of cloud-native applications with prominent services and platforms, including but not limited to Redis and PostgreSQL. Each component furnishes essential cloud-native functionalities through either automatic provisioning or standardized configuration patterns.

Developer Dashboard: Allows you to track closely various aspects of your application, including logs, traces, and environment configurations, all in real time. It’s purpose-built to enhance the local development experience, providing an insightful overview of your app’s state and structure.
Tooling/Orchestration: .NET Aspire includes project templates and tooling experiences for Visual Studio and the dotnet command-line interface (CLI) help you create and interact with .NET Aspire apps. It also provides features for running and connecting multi-project applications and their dependencies.

eShop demo application

A reference .NET application implementing an ecommerce website using a services-based architecture.

In the latest version of the application, the source code is already updated to include .NET Aspire as part of the project. You can install the latest .NET 8 SDK and clone the repository. Additionally, you can run the following commands to install the Aspire workload:

dotnet workload update
dotnet workload install aspire
dotnet restore eShop.Web.slnf

Once you have all the other prerequisites ready on your machine, you can run the application from your terminal with the following command:

dotnet run --project src/eShop.AppHost/eShop.AppHost.csproj

.NET Aspire developer dashboard

Once the application is up and running, go to the developer dashboard to identify the various resources that are part of the eShop application, including the endpoints and URL(s) to reach the running resources directly.

This dashboard also includes monitoring telemetry, including logs, traces, and metrics.

.NET Aspire orchestration

.NET Aspire provides APIs for expressing resources and dependencies within your distributed application.

Before continuing, consider some common terminology used in .NET Aspire:

App model: A collection of resources that make up your distributed application (DistributedApplication). For a more formal definition, see Define the app model.
App host/Orchestrator project: The .NET project that orchestrates the app model, named with the *.AppHost suffix (by convention).
Resource: A resource represents a part of an application whether it be a .NET project, container, or executable, or some other resource like a database, cache, or cloud service (such as a storage service).
Reference: A reference defines a connection between resources, expressed as a dependency. For more information, see Reference resources.

.NET Aspire empowers you to seamlessly build, provision, deploy, configure, test, run, and observe your cloud application. This is achieved through the utilization of an app model that outlines the resources in your app and their relationships.

Sending telemetry to an OpenTelemetry backend such as New Relic

Having a built-in dashboard for the monitoring data is nice during development. In this section I focus on how to configure OpenTelemetry correctly to send all telemetry into New Relic as my observability backend of choice.

For the scenario described in this article, I created my own fork of the official eShop application. Within this repository, you’ll be able to find the app host project that contains its main component.

Lines 17 through 26 define some basic configuration variables that you can provide using environment variables in your terminal.

NEW_RELIC_LICENSE_KEY: New Relic license key for the OpenTelemetry protocol (OTLP) API header value
NEW_RELIC_REGION: US or EU region configuration for your New Relic account

Based on the New Relic region configuration, the code will define the New Relic OTLP endpoint for OpenTelemetry and use it in the OTEL_EXPORTER_OTLP_ENDPOINT variable.

The rest of the app host project is already prepared to add an environment configuration for each of the projects that are part of the Aspire application. For example, here’s the configuration for the Identity.API project:

...
// Services
var identityApi = builder.AddProject<Projects.Identity_API>("identity-api")
    .WithReference(identityDb)
    .WithEnvironment("OTEL_EXPORTER_OTLP_ENDPOINT", OTEL_EXPORTER_OTLP_ENDPOINT)
    .WithEnvironment("OTEL_EXPORTER_OTLP_HEADERS", OTEL_EXPORTER_OTLP_HEADERS)
    .WithEnvironment("OTEL_SERVICE_NAME", "identity-api");
...

In this fork of the eShop application I’ve added some additional environment configuration. Each of the .WithEnvironment statements adds a necessary environment variable for the service:

OTEL_EXPORTER_OTLP_ENDPOINT: The OTLP endpoint for all the telemetry for this service; in our case, the New Relic OTLP endpoint.
OTEL_EXPORTER_OTLP_HEADERS: The API header value, which includes our New Relic license key (string OTEL_EXPORTER_OTLP_HEADERS = "api-key=" + NEW_RELIC_LICENSE_KEY;).
OTEL_SERVICE_NAME: The name of the service relevant to create a respective entity in New Relic.

The rest of the services are configured appropriately.

Once you’ve configured the environment variables in your terminal (that is, NEW_RELIC_LICENSE_KEY and NEW_RELIC_REGION), you can start the Aspire application with the following command:

dotnet run --project src/eShop.AppHost/eShop.AppHost.csproj

You can confirm whether everything is configured correctly by looking at the environment and clicking on the view icon for one of the projects:

The OTEL_EXPORTER_OTLP_ENDPOINT should point to the New Relic OTLP endpoint.

After a little while, you should be able to see data from your application visible in the APM & Services - OpenTelemetry section of New Relic:

You can then observe and analyze all your telemetry. For example, look at the New Relic Services map:

… or the distributed tracing view:

Happy observing!

Conclusion

Integrating OpenTelemetry with the .NET Aspire eShop application and New Relic allows you to leverage powerful telemetry tools to monitor and improve your application's performance. This setup not only provides valuable insights but also enhances your ability to diagnose issues quickly and efficiently. With the steps outlined in this guide, you're well on your way to building a more resilient and observant application. Start harnessing the full potential of your telemetry data today and keep your systems running smoothly!

Next steps

Explore more: Dive deeper into New Relic’s OpenTelemetry documentation to unlock advanced features.
Join the community: Engage with other developers on New Relic’s community forum.
Stay updated: Follow our blog for the latest tips, tutorials, and industry news.
Try New Relic for free: Sign up for a free New Relic account and start exploring how New Relic can enhance your application's telemetry today.
Experiment and iterate: Continuously monitor, analyze, and improve your telemetry setup for peak performance.

How to observe your Blazor WebAssembly application with OpenTelemetry and real user monitoring

Harry Kimpel — Mon, 10 Jun 2024 08:01:46 +0000

To read this full article, click here.

Observing WebAssembly applications presents unique challenges that stem from its design and execution environment. Unlike traditional web applications, where monitoring tools can hook directly into JavaScript and the Document Object Model (DOM), WebAssembly runs as binary code executed within the browser's sandbox. This layer of abstraction complicates direct introspection, as traditional monitoring tools are not designed to interact with the lower-level operations of WebAssembly. The Bytecode Alliance plays a crucial role here, promoting standards and tools that aim to enhance the security and usability of WebAssembly, including better support for observability. Moreover, the performance characteristics of WebAssembly, which can closely approach native speeds, demand monitoring solutions that are both highly efficient and minimally invasive to avoid impacting the user experience. This creates a complex scenario for developers who need detailed visibility into their applications' behavior without sacrificing performance.

Blazor WebAssembly: Leveraging .NET in the browser

.NET Blazor WebAssembly is a cutting-edge framework that allows developers to build interactive client-side web UIs using .NET rather than JavaScript. By compiling C# code into WebAssembly, Blazor WebAssembly empowers developers to leverage the full-stack capabilities of .NET, utilizing the same language and libraries on both the server and client sides. This unique approach streamlines the development process and enables rich, responsive user experiences with significant performance benefits.

With the release of .NET 8, Blazor WebAssembly has introduced new rendering modes that enhance flexibility and performance across diverse deployment scenarios. These modes include:

Static server rendering (also called static server-side rendering or static SSR) to generate static HTML on the server.
Interactive server rendering (also called interactive server-side rendering or interactive SSR) to generate interactive components with prerendering on the server.
Interactive WebAssembly rendering (also called client-side rendering or CSR, which is always assumed to be interactive) to generate interactive components on the client with prerendering on the server.
Interactive auto (automatic) rendering to initially use the server-side ASP.NET Core runtime for content rendering and interactivity. The .NET WebAssembly runtime on the client is used for subsequent rendering and interactivity after the Blazor bundle is downloaded and the WebAssembly runtime activates. Interactive auto rendering usually provides the fastest app startup experience.

The auto rendering mode in particular make Blazor WebAssembly an even more compelling choice for developers who are looking to build modern web applications using .NET technologies.

This blog post focuses specifically on Blazor WebAssembly and explores its capabilities and practical applications in modern web development.

Enhancing Blazor WebAssembly observability with OpenTelemetry in .NET

In this exploration of Blazor WebAssembly, we delve into how OpenTelemetry (sometimes referred to as OTel) can be integrated with .NET to provide comprehensive observability for these applications. OpenTelemetry, a set of APIs, libraries, agents, and instrumentation, allows developers to collect and export telemetry data such as traces, metrics, and logs to analyze the performance and health of applications. For .NET developers, the integration with OpenTelemetry is particularly seamless, as all telemetry for .NET is considered stable, ensuring reliability and robust support across various deployment scenarios.

Exploring a sample Blazor WebAssembly application

In this blog post, we'll be utilizing a sample application that embodies the principles of Blazor WebAssembly combined with OpenTelemetry. The application, which can be found in the GitHub repository, serves as an excellent example of a standalone Blazor WebAssembly application. This practical example will serve as the foundation for our discussion on implementing and observing OpenTelemetry in a .NET environment. By dissecting this application, we can better understand the interaction between Blazor’s client-side component as a Blazor WebAssembly application running in the browser and a Blazor WebAssembly backend application that the web frontend talks to.

Automated and manual instrumentation of the backend

When deploying OpenTelemetry in a Blazor WebAssembly application, automated instrumentation becomes a pivotal component, particularly when interfacing with the .NET backend. OpenTelemetry's .NET libraries offer out-of-the-box instrumentation for ASP.NET Core, which effortlessly captures telemetry data such as HTTP requests, database queries, and much more. This automated process simplifies the task of implementing comprehensive monitoring, as it requires minimal manual configuration and coding. Additionally, integrating OpenTelemetry into your project is as straightforward as adding the respective NuGet packages to your solution.

For developers working with Blazor WebAssembly, this means enhanced visibility into the backend operations that power their applications. Automated instrumentation ensures that all relevant data transactions between the client and server are meticulously monitored, providing insights into performance metrics and potential bottlenecks. By leveraging this feature, developers can focus more on building features and less on the intricacies of setting up and maintaining observability infrastructure, making it easier to deliver high-performance, reliable applications.

The project file for the .NET Blazor WebAssembly backend looks like this (you can find the full project file in the repository):

   <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.8.0" />
   <PackageReference Include="OpenTelemetry.Instrumentation.EntityFrameworkCore" Version="1.0.0-beta.9" />
   <PackageReference Include="OpenTelemetry.Instrumentation.Runtime" Version="1.8.0" />
   <PackageReference Include="OpenTelemetry" Version="1.8.0" />
   <PackageReference Include="OpenTelemetry.AutoInstrumentation.Runtime.Native" Version="1.5.0" />
   <PackageReference Include="OpenTelemetry.Exporter.Console" Version="1.8.0" />
   <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.8.0" />
   <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.8.1" />
   <PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.8.1" />

When running the Blazor backend with automated OpenTelemetry instrumentation and exporting the telemetry to the console, we can easily see the traces, metrics, and logs as part of the console output (here in VS Code terminal).

But looking at telemetry such as traces, metrics, and logs in the console output is not really helpful. So, ideally, we want to export that data to an OpenTelemetry telemetry backend. I am of course using New Relic.

In a typical OpenTelemetry fashion, I only need to provide a few environment variables when executing the application. I highlighted the most important ones in the below screenshot. You can find the full run script in the repository.

Once the application is up and running, you find the application entity in APM & Services under the OpenTelemetry section. The screenshot below shows the Summary view.

Distributed Tracing view:

View of a single trace (I highlighted the backend span to /roles endpoint):

When you look at the above span to the BlazorWASMBackend service GET /roles, you’ll notice that the total time spent in the backend span is 4.61s. We cannot really tell where exactly the time is spent. Out of the box, when using auto instrumentation, I don’t get further detail into what is actually happening in that method.

In order for me to provide more details into that method, I’ll need to add some manual instrumentation into my source code by leveraging the OpenTelemetry SDK for .NET.

In this case, I changed the original code of the

app.MapGet("/roles", …

method from this original implementation:

           // Instantiate random number generator using system-supplied value as seed.
           var rand = new Random();
           int waitTime = rand.Next(500, 5000);

           // do some heavy lifting work
           Thread.Sleep(waitTime);

and added in some .NET-specific implementation of a custom OpenTelemetry span:

           // Instantiate random number generator using system-supplied value as seed.
           var rand = new Random();
           int waitTime = rand.Next(500, 5000);


           using (var sleepActivity = DiagnosticsConfig.ActivitySource.StartActivity("RolesHeavyLiftingSleep"))
           {
               // do some heavy lifting work
               Thread.Sleep(waitTime);


               string waitMsg = string.Format(@"ChildActivty simulated wait ({0}ms)", waitTime);
               sleepActivity?.SetTag("simulatedWaitMsg", waitMsg);
               sleepActivity?.SetTag("simulatedWaitTimeMs", waitTime);
               DiagnosticsConfig.logger.LogInformation(eventId: 123, waitMsg);
           }

The using-block actually triggers a new span to be created with the name RolesHeavyLiftingSleep. I save the activity into the sleepActivity variable. You’ll further notice that I’m also adding some custom tags to that same activity by calling sleepActivity?.SetTag().

We can see in the screenshot below what the result of that change looks like if we restart our application.

The trace in the screenshot above allows me to drill deeper into the backend span by enabling the in-process spans. In here, we can see that there’s a RolesParentActivity, followed by a RolesChildActivity, and finally our RolesHeavyLiftingSleep span from our manual instrumentation. This screenshot also shows a section of the attributes that are associated with that span. As you can see, the custom tags simulatedWaitTimeMs and simulatedWaitMsg are visible as well. These custom tags are potentially helpful when doing some root cause analysis or troubleshooting.

Instrumentation of the frontend

Now that we’ve seen how the backend can be instrumented with OpenTelemetry, let’s focus on the frontend now, that is, the Blazor WebAssembly component.

When we try to instrument the WebAssembly component with auto instrumentation for OpenTelemetry, the C# project file looks similar to the backend. This is actually the benefit of Blazow WebAssembly—that we can use C# not only for the backend, but also for the WebAssembly frontend.

Let’s configure the console exporter again for our OpenTelemetry telemetry. Unsurprisingly, the output will be visible in the respective developer tools of our web browser.

The screenshot above shows the actual UI of the Blazor WebAssembly frontend in the upper section of the screenshot. If the user of the application clicks the Click me button, a trace is generated and output in the browsers console view.

Again, this is adequate for testing, but in order to actually leverage the telemetry in a meaningful way, we need to export all telemetry to an OpenTelemetry backend, in my case New Relic. For this to happen, we need to define an OpenTelemetry protocol (OTLP) exporter and configure the OTLP endpoint as well as the OTLP header information similar to what we’ve seen for the backend.

But wait; as soon as we implement these changes and rerun the application, an exception is generated.

The exception is Operation is not supported on this platform. If we think about it, it does make sense since we’re trying to make a HTTP request from our WebAssembly component to an external endpoint. However, since WebAssembly by itself doesn't have any access to its host environment, it doesn't have any built-in input/output (I/O) capabilities. For security reasons, it’s not possible to make HTTP requests from within a WebAssembly component.

So, if plain OpenTelemetry instrumentation isn’t possible, what is it that we can do for the frontend then?

As part of OpenTelemetry, the community is also working on some real user monitoring capabilities.

However, this is very early stages and not fully spec’d out. The current draft is also only focusing on Node.js and TypeScript. In the future this may be an option that could be leveraged for the frontend component.

One way to get details of the WebAssembly component is to leverage real user monitoring capabilities via the New Relic browser.

After getting the JavaScript snippet from a newly created New Relic browser application copied into the Blazor WebAssembly project (this is the place to put it), we can already see some high-level telemetry from the frontend.

Distributed tracing view:

AJAX requests:

What else can we do with the frontend? Some parts of the frontend will have interactions with the backend (like login and logout authentication). Other parts just execute code within the WebAssembly component. An example of this is the Counter section.

As you can see in the page source of that page, there’s no actual HTML representation.

Let’s see what we can do there.

One way is to invoke JavaScript functions from the actual .NET code. The following screenshot shows how this can be achieved (here is the link to the respective file in the repository).

The New Relic browser API allows users to add some custom page actions. This way we can observe all clicks on the counter and also capture the current value of the counter as a custom attribute.

Once we’ve implemented this and deployed a new release of the application, we can for example show this data on a custom dashboard to see the distribution of the actual counter values across all users of the application.

Furthermore, New Relic quickstarts, also known as Instant Observability, contain a sample dashboard that you can deploy into your account in order to see some additional Blazor WebAssembly specific telemetry in a pre-built dashboard.

Next steps

Observing Blazor WebAssembly applications is not quite straightforward as of today. There are many moving parts that the industry and the respective open-source communities are working on. This applies to the WebAssembly component model, as well as the OpenTelemetry implementation of real user monitoring. I think these challenges will soon be solved and there will be easier ways to get started.

Until then, in this blog post I showed a way to get some insights into your Blazor WebAssembly backend and frontend components.

New Relic provides a free account that you can use to get started with your journey on observing Blazor WebAssembly applications.

Guide: How to route Docker logs correctly in New Relic

Harry Kimpel — Fri, 12 Apr 2024 05:36:57 +0000

To read this full article, click here.

Streamlining Container Log Management for Clarity and Control

Hello, New Relic aficionados! Picture this: you're at a bustling local user group meetup, exchanging ideas and sharing tech stories. Amidst the animated discussions and clinking coffee cups, a fellow developer—let’s call him Alex—shares a frustrating puzzle. Alex’s Docker Compose applications are acting like rebellious teenagers, sending their logs to the New Relic Host UI instead of their designated New Relic Container UI. As you dive deeper into the problem, a light bulb goes off. This isn't just Alex's struggle; it’s a common snag affecting many of us in the Docker and New Relic community.

Why do these logs prefer the scenic route, and how can we guide them to their proper home? Inspired by this real-life challenge, I embarked on a quest to demystify the log misdirection issue and share a roadmap to log management nirvana with New Relic. So, grab your digital compasses, dear readers—it's time to navigate through the foggy waters of Docker logging and ensure your data logs exactly where it should.

Understanding the core issue: Where do my container logs end up?

So, what's the fuss about where logs end up anyway? Let's break it down. When you're running applications in containers, especially when using Docker, logging isn't just about keeping a record; it's about clarity and context. The New Relic infrastructure agent (along with the New Relic integration for Docker) is designed to be thorough—it dutifully collects logs from the host and any containers running on it. But here’s the snag: instead of neatly categorizing container logs under each container, all logs get lumped together. Your container logs are showing up right alongside host/server logs in the New Relic infrastructure Hosts UI.

And the Logs section of your Container UI does not show any logs:

Why does this matter? Imagine trying to find a specific conversation in a bustling, crowded room. Every discussion, whether it’s crucial or trivial, merges into one overwhelming cacophony. That's your current logging scenario—container logs are getting lost in the noise of the host logs, making it challenging to pinpoint issues or understand the behavior of specific containers.

This misassignment doesn’t just clutter your view; it complicates monitoring and troubleshooting by obscuring the boundaries between container-specific operations and overall host activity. You need to see your container logs isolated from host logs to quickly diagnose issues, scale effectively, and understand exactly how your containers are performing in the wild.

In the following sections, we'll explore why this log misplacement occurs and how you can reroute your logs to their correct destinations in New Relic, ensuring that your monitoring setup is as sharp and efficient as your development environment.

Configuring your environment for precise log routing

Now that we understand the issue at hand, let's roll up our sleeves and tackle the solution. Organizing your logs with New Relic starts with a few key adjustments in your environment. Here’s a simple guide to ensure your container logs aren’t just collected but correctly associated with their respective container entities in New Relic Logs.

Step 1: Adjust your logging configuration

Chances are, this step might already be in place if you've been monitoring with New Relic. Your goal here is to ensure that the logs from each container are being captured effectively. This involves tweaking your logging configurations to include detailed information about each container's operations. If this isn't set up yet, here's what you typically need to modify in your environment settings to start capturing those logs.

Here’s the snippet that you can copy and use in your logging.yml:

- name: containers
    file: /var/lib/docker/containers/*/*.log

Step 2: Tagging containers in Docker

Next, you’ll need to make your containers recognizable by tags when running your containers manually using ‘docker run’ command or within your Docker Compose configuration file. Adding the below tag makes this happen:

Docker run command example

docker run --log-opt tag="{{.Name}}" -d log-generator

Docker Compose example in docker-compose.yml

Here’s the snippet that you can copy and use in your docker-compose.yml:

version: '3.9'
x-default-logging: &logging
  driver: "json-file"
  options:
    max-size: "5m"
    max-file: "2"
    tag: "{{.Name}}"

This configuration ensures that each log entry from your containers includes the container's name as a tag, making it easier to track in the logs collected by New Relic.

Step 3: Set up a log parsing rule in New Relic Logs

Once you start seeing the container tags coming through in your New Relic environment, it’s time to refine how these logs are parsed and displayed. Setting up a log parsing rule helps in categorizing and querying logs based on specific attributes.

Here’s how to configure it:

Name: container name
Field to parse: attrs.tag
Filter logs based on New Relic Query Language (NRQL): attrs.tag is not null
Parsing rule: %{NOTSPACE:containerName}

This parsing rule will extract the container name from the attrs.tag field of each log entry and add an additional attribute named containerName, enabling you to filter and analyze logs by specific containers.

Conclusion

With these three steps, your container logs will not only be more organized but also more insightful. By ensuring that each log entry is correctly tagged and parsed, they'll directly feed into New Relic's powerful logging tools, allowing you to monitor and troubleshoot with unmatched precision. This setup not only enhances your ability to respond to issues swiftly but also empowers your team with data-driven insights, ensuring your applications run smoothly and efficiently.

As we wrap up our guide on streamlining Docker logs in New Relic, remember that effective log management is just one piece of the observability puzzle. To further illustrate the power of integrated monitoring tools, I recently contributed to enhancing the OpenTelemetry demo application. My pull request (PR #1495) has been accepted and merged, introducing a new tagging feature that you'll find particularly useful when running the demo with Docker Compose. This addition helps ensure that container logs are easily distinguishable and accurately attributed, enhancing your ability to monitor and troubleshoot effectively. Check out the changes and explore how they can benefit your setup here.

In addition to contributing to the OpenTelemetry demo application, I've also updated the respective sections of the New Relic documentation to reflect these improvements in log management. This update ensures that the strategies and technical details we've discussed are not only tested but also officially integrated into our guidance, making it easier for you to implement and benefit from these changes. You can view these updates (soon) to enhance your understanding and application of these practices, ensuring you get the most out of your New Relic tools.

Next steps

Congratulations on configuring your Docker environment for optimal log management! But the journey doesn’t stop here. Dive deeper into the possibilities:

Experiment and refine: Adjust the logging levels and parsing rules based on the specific needs of your environment. Each application may have unique insights to offer.
Monitor the impact: Keep an eye on how these changes enhance your monitoring capabilities. Look for improvements in troubleshooting and system performance.
Share your insights: Got a handle on things? Don’t keep it to yourself. Share your success stories and challenges in the comments below or on social media. Your experiences could shine a light for fellow developers navigating similar challenges.
Engage with the community: Join discussions on the New Relic Explorers Hub to connect with other users and exchange tips and tricks. Your feedback not only contributes to community growth but also helps us improve and evolve our solutions.

Ready to elevate your log management game? Start tweaking, sharing, and engaging today. Let’s harness the full potential of precise log data together!

A deep dive into zero-day vulnerability alerts with New Relic APM

Harry Kimpel — Fri, 23 Feb 2024 18:52:42 +0000

To read this full article, click here.

Amidst the ever-evolving landscape of cybersecurity, the recent revelation of a zero-day vulnerability in Fortinet's FortiOS serves as a stark reminder of the constant cat-and-mouse game between defenders and attackers.

Staying ahead of potential security threats isn’t just a best practice; it's a necessity. For developers, the challenge lies not only in identifying vulnerabilities but in doing so proactively, especially when it comes to zero-day exploits. In this blog post, we'll explore how New Relic application performance monitoring (APM) empowers developers to create zero-day vulnerability alerts, offering a robust solution to enhance security postures without the need for extensive scanning.

Developers are often tasked with managing the delicate balance between agility and security. New Relic recognizes this challenge and provides a comprehensive set of tools to streamline the process. Today, we'll delve into two key capabilities—alert conditions and policies within the New Relic platform and the integration with Vulnerability Management—that enable developers to create targeted alert rules and effortlessly control their security posture.

Let's embark on a journey through these capabilities, exploring how they equip developers to receive timely notifications on specific common vulnerabilities and exposures (CVEs) and maintain an up-to-date understanding of their application's security status. Additionally, we'll unravel the magic of the New Relic Database (NRDB) and the Environment Snapshot tab, where all changes, including library modifications, are meticulously recorded. Join us as we navigate the realm of zero-day vulnerability alerts, unlocking the full potential of New Relic APM for developers.

What exactly are zero-day vulnerability alerts?

Before we dive deeper into the advantages of zero-day alerts, let's understand what zero-day alerts are.

Zero-day alerts are crucial in software development. Picture this: You're in the midst of coding, and suddenly, an alert pops up—a zero-day vulnerability is detected. It's not just any vulnerability; it's one that nobody knew about until now. Zero-day alerts are like unexpected guests—they demand immediate attention.

These alerts signal the emergence of previously unknown vulnerabilities or security threats. Unlike known issues with patches, zero-day vulnerabilities are wild cards, demanding swift action and vigilance.

Some developers may think that once their source code has been scanned for vulnerabilities at build time, their job is done. However, once the application or service is in production, on average up to three years later, a vulnerability exposure will be disclosed that was not known when the source code was originally scanned. Finding and fixing these vulnerabilities is often a crisis moment for many organizations.

Zero-day alerts aren't just notifications; they're urgent calls to action. They remind us of the ever-changing digital landscape, urging us to stay vigilant and responsive in our defenses. They're about staying ahead, anticipating the unexpected, and protecting our digital realms from the unknown. In software development, they're the plot twists that keep us on our toes, ready to tackle surprises head-on.

Unveiling the developer advantages

As developers, embracing a proactive security posture is not just a choice; it's a strategic advantage. New Relic APM and Vulnerability Management provide a dynamic duo that equips development teams with an array of benefits, revolutionizing the way we approach security in the software development lifecycle.

Here are some benefits of this approach:

Real-time alerts on zero-day events:
Traditional scanners operate on scheduled scans, often missing the critical moment when a zero-day library event occurs. With APM and Vulnerability Management, developers receive real-time alerts, ensuring swift responses to potential vulnerabilities. This capability surpasses the limitations of scanners, providing a level of immediacy crucial in today's fast-paced development landscape.
Broad visibility across your entire environment:
The APM agent acts as a vigilant sentinel, offering broad visibility across your entire environment. It goes beyond individual applications, providing insights into what's running, where it's running, and the security status across thousands of applications. This holistic perspective empowers developers with a comprehensive understanding of their application landscape, surpassing the capabilities of traditional scanning tools.
Instant impact assessment:
Imagine having immediate answers to critical questions: Where are we affected? What is the impact? How do I fix it? APM and Vulnerability Management not only provide alerts but also enable developers to assess the impact instantly. Automation further streamlines the process, allowing for quick decision-making and efficient remediation. Developers can stay informed about the status of remediation efforts assigned to specific problems, fostering a culture of accountability and transparency.
Continuous zero-day analysis:
While scanners offer point-in-time snapshots that may become outdated with changes to code bases or environments, APM and Vulnerability Management provide continuous analysis. Every change in your environment is captured and assessed in real time. This ensures that your security posture isn’t just a momentary snapshot but an ongoing, adaptive process that evolves with your application.
Proactive prevention and collaboration:
APM and Vulnerability Management go beyond mere detection; they empower developers to proactively prevent issues. Receive notifications to avoid calling a specific library due to the need for an upgrade, preventing the creation of a stack of baseline library vulnerabilities associated with multiple entities. This proactive approach not only mitigates risks but also minimizes the high cost of addressing vulnerabilities down the road.
Automatic detection and fixing of new dependencies:
Stay ahead of the curve by automatically detecting and fixing vulnerabilities in new dependencies. APM and Vulnerability Management enable developers to address issues before they propagate, often resolving vulnerabilities before the security team is even aware of the issue. This level of automation not only enhances security but also optimizes development workflows, allowing teams to focus on innovation rather than firefighting.

Now, armed with the knowledge of these substantial benefits, let's explore the practical steps to harness these capabilities and create zero-day vulnerability alerts using New Relic APM, alerts and applied intelligence.

Creating targeted alert rules with New Relic

As developers, vigilance is key when it comes to security, and New Relic APM provides a powerful ally in this pursuit. With the alerts and applied intelligence capabilities, developers can seamlessly create tailored alert rules to receive timely notifications on specific CVEs.

Imagine a scenario where a critical CVE is identified, and swift action is necessary. New Relic APM allows you to navigate to the Alert Conditions tab, where you can set up customized conditions based on specific parameters such as error rates, response times, or throughput. By integrating Vulnerability Management into this process, you can extend your alert rules to cover vulnerabilities, making your security response not just rapid but also finely tuned to the unique characteristics of your application.

Let's break it down:

Navigate to Alerts & AI in the New Relic platform:

Locate the Alert Policies section to access the powerful alerting capabilities of the New Relic platform.

Create a New Alert policy:

Define a new alert policy tailored to your application's needs. This policy will serve as the foundation for your customized alert rules.

Create an alert condition

Provide a name “New Zero Day Library Vulnerability”
Enter the New Relic Query Language (NRQL) query

SELECT count(*) FROM Vulnerability where issueType = 'Library Vulnerability'
Define thresholds. The important aspect in this section is “Open incidents with a:”. Here you’ll specify when to trigger a critical incident. Of course, we want to get alerted as soon as the query returns a value above 0 at least once in the given time window.
Add details. Provide a name for your alert condition and adjust the other settings as you see fit.

By following these steps, you empower your development team with a proactive stance against vulnerabilities, receiving notifications that are not just timely but also precisely aligned with your application's unique security requirements.

Next steps

Next, let's explore how New Relic goes beyond alerting by providing a comprehensive record of all changes, ensuring a thorough understanding of your application's security landscape.

To make it even easier for you to get started, I’ve created a GitHub repository that contains a Terraform script to create a sample alert policy and condition using the above concept. Alternatively, it also contains a NerdGraph query (New Relic's GraphQL API) that you can use along with New Relic’s New Relic's Graphiql Explorer.

Get started with your free New Relic account today.

How to monitor Microsoft 365: Observing AD FS

Harry Kimpel — Sun, 26 Nov 2023 23:00:00 +0000

To read this full article, click here.

A practical guide to Active Directory Federation Services for a resilient Microsoft 365 ecosystem

In our previous blog on how to monitor Microsoft 365 (M365), we delved into service overviews and the critical importance of synthetic user login monitoring. In this blog, we set our sights on a core component that forms the backbone of secure identity and access management: Active Directory Federation Services (AD FS).

As organizations increasingly migrate their operations to the cloud, ensuring the robustness of identity and authentication mechanisms becomes paramount. AD FS plays a pivotal role in this landscape, acting as the linchpin for seamless and secure single sign-on (SSO) experiences within the M365 ecosystem.

In this installment, we aim to equip you with the knowledge and tools needed to ensure the reliability and security of your AD FS implementation: navigate the terrain of certificate validations and metadata exchange documents, and unravel key elements that warrant vigilant oversight in your M365 environment Let's dive into the world of AD FS and uncover the essentials of effective monitoring.

Validate AD FS certificates

Validating AD FS certificates is a crucial aspect of maintaining a secure and reliable authentication infrastructure within M365. Certificates serve as cryptographic keys that facilitate secure communication between different components of the AD FS environment. Regular validation ensures that these certificates are not only genuine but also up to date, reducing the risk of unauthorized access or security breaches. An expired or compromised certificate can lead to service disruptions, hindering the seamless flow of authentication requests. By enforcing rigorous certificate validation practices, organizations can fortify their AD FS implementation, enhance overall security, and provide users with a consistent and trustworthy SSO experience.

Follow the steps below to get started with monitoring your AD FS certificates.

AD FS monitoring is implemented as an on-host integration for the New Relic infrastructure agent. All of the configuration and necessary scripts are provided in a dedicated GitHub repository.

This integration will typically run on the same server that hosts the AD FS role. In order to get the integration deployed, follow these steps:

Install the New Relic infrastructure agent (if you need assistance, follow the guided install).
Copy the configuration file adfs-cert.yml and the PowerShell script GetExpiringCertificates.ps1 into the agent’s integration folder. These are the default locations:

Linux: /etc/newrelic-infra/integrations.d/
Windows: C:\Program Files\New Relic\newrelic-infra\integrations.d

Restart the infrastructure agent service.

The configuration file is straightforward and looks like this:

integrations:
- name: nri-flex
   interval: 24h
   config:
     name: M365AdfsCertificate
     apis:
       - event_type: M365AdfsCertificate
         shell: powershell
         timeout: 299000
         commands:
           - run: '& "C:/Program Files/New Relic/newrelic-infra/integrations.d/GetExpiringCertificates.ps1"'

Line 7 in the above configuration defines the name of the event where the data from this integration is stored in New Relic. We instruct the infrastructure agent to leverage the Flex integration (line 2) to leverage a PowerShell shell (line 8) in order to call the script defined in the run command (line 11).

The actual PowerShell script looks like this:

$expiring_certs = Get-ChildItem -Path cert: -Recurse -ExpiringInDays 365  | Select-Object Issuer, NotBefore, NotAfter, Subject, FriendlyName, SerialNumber, Thumbprint

# Build an empty array to add our results to
$results = @()

$StartDate = (Get-Date)

foreach ($item in $expiring_certs) {

 $ts = New-TimeSpan -Start $StartDate -End $item.NotAfter
 $tsDaysReverse = $ts.Days * -1

  # Build a custom object to pass into the results
 $cert = [ PSCustomObject ]@{
   certSubject               = $item.Subject
   certIssuer                = $item.Issuer
   certSerialNumber          = $item.SerialNumber
   certNotBefore             = ( [DateTimeOffset ]$item.NotBefore ).ToUnixTimeSeconds()
   certNotAfter              = ( [DateTimeOffset ]$item.NotAfter ).ToUnixTimeSeconds()
   certThumbprint            = $item.Thumbprint
   certExpiringIn            = $ts
   certExpiringInReverseDays = $tsDaysReverse
   certExpirationDate        = $item.NotAfter | Get-Date -Uformat %s
   certFriendlyName          = $item.FriendlyName
 }

 $results += $cert
}

$results | ConvertTo-Json

The first thing the script does is leverage the Get-ChildItem module to get all certificates that are expiring in the next 365 days. Next, we construct an empty array that will be returned at the end. In a loop, we create new objects for each certificate found and add it including all the details to the results array. The final array will be converted into JSON and returned as output of the script.

In the New Relic UI, we use the entity explorer to look at all the raw data that’s being collected.

We can also build a custom dashboard to visualize the data in a meaningful way.

Although we can refer to the data and dashboard this isn’t something that I want to manually check from time to time. Ideally, I want to get an alert notification if, for example, there’s a certificate about to expire in the next 30 days. This would probably give me enough time to renew a certificate or create a new one. With New Relic, this can easily be done by setting up an alert condition using a New Relic Query Language (NRQL) query.

SELECT min(certExpiringIn.Days) as 'Cert expiring' from M365AdfsCertificate facet certSubject

In the threshold configuration, I can specify to trigger an incident whenever that query returns a value below 30.

Availability of the metadata exchange document

Ensuring the availability of the metadata exchange document is paramount for maintaining a resilient AD FS infrastructure within the M365 environment. The metadata exchange document contains critical information about AD FS endpoints, certificates, and other key metadata necessary for secure communication and authentication. Regularly checking its availability is essential to guarantee that this vital information is readily accessible to federation partners and other components in the ecosystem. An unavailable metadata exchange document can disrupt the federation process, leading to authentication failures and potential service outages. Proactively monitoring its availability allows organizations to identify and address issues promptly, ensuring the uninterrupted flow of authentication data and contributing to a robust and reliable M365 experience for users.

Follow the steps below to get started with monitoring your metadata exchange document.

AD FS monitoring is implemented as an on-host integration for the New Relic infrastructure agent. This integration will typically run on the same server that hosts the AD FS role. To deploy this integration, follow these steps:

Install the New Relic infrastructure agent (if you need assistance, follow the guided install). Note: If you already followed the steps described above on validating certificates, you can skip this step and start with step 2.
Copy the configuration file adfs-metadata-xml.yml and the PowerShell script GetMetadataXML.ps1 into the agent’s integration folder. These are the default locations:

Linux: /etc/newrelic-infra/integrations.d/
Windows: C:\Program Files\New Relic\newrelic-infra\integrations.d

Restart the infrastructure agent service

Again, the configuration file is straightforward and looks like this:

integrations:
- name: nri-flex
   interval: 60m
   config:
     name: M365AdfsMetadata
     apis:
       - event_type: M365AdfsMetadata
         shell: powershell
         timeout: 299000
         commands:
           - run: '& "C:/Program Files/New Relic/newrelic-infra/integrations.d/GetMetadataXML.ps1"'

The PowerShell script looks like this:

add-type @"
   using System.Net;
   using System.Security.Cryptography.X509Certificates;
   public class TrustAllCertsPolicy : ICertificatePolicy {
       public bool CheckValidationResult(
           ServicePoint srvPoint, X509Certificate certificate,
           WebRequest request, int certificateProblem) {
           return true;
       }
   }
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

$metadataUrl = "https://localhost/FederationMetadata/2007-06/FederationMetadata.xml"
$result = Invoke-WebRequest $metadataUrl -UseBasicParsing

# Build a custom object to pass into the results

$jsonResult = [ PSCustomObject ]@{

   metadataXMLURL    = $metadataUrl
   statusCode        = $result.StatusCode
   statusDescription = $result.StatusDescription
   rawContentLength  = $result.RawContentLength

}

$jsonResult | ConvertTo-Json

In line 14 we define the URL of the metadata exchange document which we then pass into the Invoke-WebRequest function (line 15). Next, we analyze the result and create an object with some details about the metadata exchange document, including the results from the web request; that is, whether or not the request was successful.

In the New Relic UI, we use the entity explorer to look at all the raw data that’s being collected.

We can also build a custom dashboard to visualize the data in a meaningful way.

As we’ve seen in the previous example with expiring certificates, I want to take the proactive route and have New Relic alert me whenever a metadata exchange document is no longer available. With New Relic, this can easily be done by setting up an alert condition using an NRQL query.

SELECT latest(statusCode) FROM M365AdfsMetadata where metadataXMLURL is NOT NULL facet metadataXMLURL

The above query returns the latest status code for each of the metadata exchange documents that I’m monitoring. If for any of these paths a status code of 400 or above occurs, I want to get an incident triggered.

Conclusion

Now that we've unraveled the intricacies of AD FS monitoring, it's time to empower your organization with a robust solution. By following the steps outlined in this blog, you can enhance your monitoring capabilities and fortify your Microsoft 365 environment.

Next steps

Take the proactive step towards a more secure and reliable Microsoft 365 experience. Create your free New Relic account today and unlock a new era of AD FS monitoring excellence.

How to use CodeStream—and shift left your observability practice

Harry Kimpel — Tue, 31 Oct 2023 19:00:00 +0000

To read this full article, click here.

Your adventure in the world of observability, performance optimization, and security begins here.

In the fast-paced world of software development, the quest for smoother, more efficient applications feels like a never-ending adventure. But in this adventure, it's not about slaying dragons or uncovering hidden treasures; it's about ensuring your code performs seamlessly and your users are happy.

As a developer, I've faced the challenges of reactive observability, where we typically identify and resolve issues only after they've disrupted our applications. It's a scenario that's all too familiar for many of us. Adding New Relic CodeStream to your tool inventory can help rescue your team from a perilous fate and bring their observability game to a new level.

Picture this: you're deep into a new project, and your code is flowing like water. You're in your element, coding away. But at the back of your mind there's always that nagging concern: Will this code run smoothly in the wild, or are there lurking issues you've yet to uncover?

Enter New Relic CodeStream, a game-changer that's become my trusty companion in the world of software development. What's different about this tool? It's about shifting observability left in your development journey.

Understanding the shift-left approach

Before we dive into how New Relic CodeStream can help you shift left with observability, it's important to understand the concept of shifting left itself. Traditionally, observability has been associated with monitoring and debugging in production. The shift-left approach encourages you to bring observability into the development and testing phases of your software development process. This proactive approach allows you to identify and address performance and reliability issues early in the development cycle, reducing the risk of deploying problematic code.

A game changer for shifting left

New Relic CodeStream is a developer-friendly observability platform designed to seamlessly integrate observability into your existing workflows. It allows developers to collaborate, visualize, and diagnose issues within their preferred development tools.

In this blog, I’m using a fork of a replica of the 1988 DOS game Dangerous Dave. This provides a fun way for me to showcase my adventure. And, who doesn’t like to play a good game from time to time. ;-) The purpose of this game is to collect as many items as possible in order to increase my score and finish as many levels as possible.

Here's how you can use CodeStream to shift left with observability:

Collaborative observability

Collaboration is at the heart of this shift-left approach. With CodeStream, I no longer feel like I'm navigating the code wilderness alone. I can discuss, share insights, and even create tasks within my development environment. It's like having a team of fellow adventurers by your side, ready to help you conquer any challenges.

CodeStream does that by providing telemetry data at the service level, including all related services, to present a general snapshot and overview of what I’m currently working on.
The ability to drill down and get to the golden signals at the code level provides me with more information to understand and improve my code. This integration of observability into my daily practice right inside my favorite IDE makes me so much more efficient.

I even have all the high level information at hand for my application in order to make sure that the golden signals for my application look healthy. Being able to get such insights into the IDE allows me to stay focused and not have to switch context from my deep development tasks.

In the example below, I’m discussing with another user an idea on how to improve observability insights into the game. Again, I’m not required to switch to a different tool, make screenshots, or copy and paste code-snippets. I have everything right where it belongs from the perspective of a developer.

Early issue detection

CodeStream surfaces performance metrics next to code to speed up error resolution and streamline planning and implementing performance improvements while simplifying code collaboration.

It’s a huge benefit, because I'm not waiting for issues to show up in production; I'm being alerted as soon as there's a deviation from the expected. It's like having a scout who lets you know when you're veering off course.

In my case—as you can see in the below screenshot—everything looks good and I can happily continue playing my game.

However, in another example (this time from a different application), things have taken a turn for the worse and the road ahead looks perilous. I see some extreme increase in both error rate as well as duration of different transactions within my application. This is an important indicator for me to swarm with my team and run some troubleshooting sessions in order to get this back to normal.

Root cause analysis

Let’s face it, bugs happen. When the inevitable issue arises, CodeStream has my back with its powerful tools for root cause analysis. I can view information about issues along with the entire stack trace in one unified interface. It's like having a treasure map that guides you to the heart of the issue, saving valuable time and resources. It helps us navigate the complex landscapes of issues efficiently, ensuring that observability is not a distant concept but a part of our entire development journey.

New Relic CodeStream provides an Errors section that easily allows me to look at all the issues assigned to me so that I can keep track of my to-do list. Additionally, I can also look at other more recent errors that occurred.

From there I can dive deeper into that ZeroDivisionError seen in the previous screenshot. As you can see in the next screenshot, CodeStream provides all relevant information that I need to get to the root cause of the issue. I can click into the first item in the stacktrace…

And follow through to the next item…

Until I reach the actual line of code where the exception occurred.

It’s a little like having the cheat codes, right?!

Vulnerability insights at your fingertips

But there's more to this adventure than just performance and reliability. With the New Relic comprehensive platform, you have a vigilant guardian at your side. The New Relic platform is designed to uncover vulnerabilities before they become critical threats. It identifies security issues and provides invaluable insights, all seamlessly integrated into your development environment through New Relic CodeStream.

Imagine having a security wizard working alongside you as you code. The New Relic platform keeps a watchful eye on your codebase, scanning for vulnerabilities and pinpointing security weak points. It's like having a trusted advisor who ensures that your applications are fortified against potential threats.

This integrated approach allows you to stay informed about the latest security threats without ever leaving your preferred integrated development environment (IDE). New Relic CodeStream provides you with the security insights you need to protect your applications, all within the familiar and convenient interface of your development environment.

I can see each of the libraries that are affected by known vulnerabilities.

And can even learn more about a single vulnerability right within my IDE.

Seamless integration

One of the things I love about CodeStream is how seamlessly it fits into my existing workflow. It's like adding a new piece to your armor; it doesn't weigh you down, but it provides an extra layer of protection. I can access New Relic data and insights without leaving my development environment, making observability an integral part of my daily routine.

Additionally, I can share data and more information about other tools I use on a daily basis, and, more importantly, that are also used by other team members in my organization.
For example, I can easily create comments into messaging tools such as Slack or Microsoft Teams. Or create an issue and share it via any of my favorite tools.

For example, via a GitHub issue.

Which I can then review and track as part of my natural workflow in GitHub itself.

The ongoing quest

Shifting left with observability isn't a one-time adventure; it's a continuous quest for improvement. With the New Relic platform and CodeStream, you can not only track your application's performance over time but also bolster its security defenses. It's data-driven decision-making at its best, helping you evolve and adapt your applications to meet the ever-changing user demands while staying one step ahead of potential vulnerabilities.

Conclusion

In the world of software development, where every line of code can make or break an application, the New Relic platform and CodeStream are the beacons that guide us. They're not just tools; they're trusted companions on our journey to create robust, high-performance, and secure applications. So, heed the call, embark on the adventure, and use the New Relic platform and CodeStream to revolutionize your observability and fortify your application's defenses. Your users will be the ones to tell the tales of your legendary, reliable, and secure applications.

Next steps

As you embark on your journey to shift left with observability and embrace the power of New Relic CodeStream, why not experience it firsthand? New Relic offers a free account to get you started on your path to creating high-performance, reliable, and secure applications. Take the first step, sign up for your free account, and witness the transformation of your development process.

Your adventure in the world of observability, performance optimization, and security begins here. With New Relic CodeStream as your trusty companion, you're ready to conquer any challenge that comes your way. Sign up today, and let's make your applications legendary.

How to monitor Microsoft 365: Getting started

Harry Kimpel — Tue, 17 Oct 2023 18:34:35 +0000

To read this full article, click here.

One of the most popular and widely adopted business tool suites is Microsoft 365 (previously known as Microsoft Office 365) or just M365 for short. M365 offers a comprehensive array of applications and services, including email, document collaboration, video conferencing, and more, all hosted on the cloud. M365 provides numerous benefits in terms of flexibility and accessibility, but it also introduces new security, performance, and compliance challenges. This is why monitoring M365 applications is critically important for organizations of all sizes. Monitoring M365 applications is essential in helping businesses maintain a secure, efficient, and compliant digital environment.

In this blog, we’ll look into various parts of the M365 ecosystem. We begin our monitoring journey by gathering information about the health status of the services used within your subscription. We’ll then dig a little deeper into real-time user login monitoring.

Health information of your services subscribed by a tenant

In M365 adding agents for monitoring isn't possible. Microsoft offers M365 as a software as a service (SaaS); hence, you don’t have access to the underlying infrastructure and cannot install any language or infrastructure agents. To collect health metrics, we have to rely on native Microsoft tools and APIs by utilizing features like the Health Dashboard, auditing, reporting, and the Microsoft Graph API to track and analyze M365 performance, security, and user activity. This ensures a reliable digital workplace without third-party monitoring agents.

As part of Microsoft Graph REST APIs, we’re able to gather information about the health information of services subscribed by a tenant.

An easy way to gather this information on a continuous basis is to run a New Relic synthetic scripted API monitor.

Follow these steps to get started:

Create a new synthetic monitor and then select Endpoint availability (Scripted API) as the monitor type.

Select your account and provide a name of the monitor; for example, Microsoft 365 - service overview. Period should be set to the interval on how often you want to get the data for the service health.

Click Select locations and select a single location (in our case it doesn’t make sense to provide additional information when running the API call from many locations).
Continue to the Write script section. Here we’ll write the API test for the service health overview. Copy and paste the below script content into the Script editor.
Click Save monitor.

/**
 * Feel free to explore, or check out the full documentation
 * https://docs.newrelic.com/docs/synthetics/new-relic-synthetics/scripting-monitors/writing-api-tests
 * for details.
 */

var assert = require('assert');
const request = require("request");
const { promisify } = require("util");
const promisifyRequestGet = promisify(request.get);
const promisifyRequestPost = promisify(request.post);

/**
 * VARIABLE DEFINITIONS
 */
var MSFT_GRAPH_ACCESS_TOKEN = $secure.MSFT_GRAPH_ACCESS_TOKEN;
var NEW_RELIC_ACCOUNT_ID = $secure.NEW_RELIC_ACCOUNT_ID;
var NEW_RELIC_INSIGHTS_INSERT_KEY = $secure.NEW_RELIC_INSIGHTS_INSERT_KEY;
var MSFT_TENANT_ID = $secure.MSFT_TENANT_ID;
var MSFT_CLIENT_ID = $secure.MSFT_CLIENT_ID;
var MSFT_CLIENT_SECRET = $secure.MSFT_CLIENT_SECRET;
var MSFT_USERNAME = $secure.MSFT_USERNAME;
var MSFT_USER_PASSWORD = $secure.MSFT_USER_PASSWORD;
var NEW_RELIC_EVENT_TYPE = 'M365ServiceOverview';

/**
 * Function to post events to the New Relic Events API
 * @param {*} body 
 * @returns {Promise<request.Response>}
 */
async function insertInsightsEvent(body) {
    return await promisifyRequestPost({
        uri: 'https://insights-collector.newrelic.com/v1/accounts/' + NEW_RELIC_ACCOUNT_ID + '/events',
        body: body,
        headers: {
            'X-Insert-Key': NEW_RELIC_INSIGHTS_INSERT_KEY,
            'Content-Type': 'application/json'
        }
    });
}

/**
 * Function to record data in NRDB.
 * @param events
 * @returns {Promise<void>}
 */
async function recordData(events) {
    //console.log(events);
    let body = JSON.stringify(events, null, 2);
    const insightsResponse = await insertInsightsEvent(body);
    if (insightsResponse.statusCode !== 200) {
        console.log("insertInsightsEvent() non-200 return code: " + insightsResponse.statusCode);
    }
    else {
        console.log('Script executed successfully');
    }
}

var urlAuthBody = '';
urlAuthBody += 'client_id='+MSFT_CLIENT_ID;
urlAuthBody += '&client_secret='+MSFT_CLIENT_SECRET;
urlAuthBody += '&response_type=token';
urlAuthBody += '&scope=ServiceHealth.Read.All%20user.read%20openid%20profile%20offline_access';
urlAuthBody += '&username='+MSFT_USERNAME;
urlAuthBody += '&password='+MSFT_USER_PASSWORD;
urlAuthBody += '&grant_type=password';

var options = {
    body: urlAuthBody,
    headers: {
        'Content-Type': 'application/x-www-form-urlencoded'
    }
}

var urlAuth = 'https://login.microsoftonline.com/' + MSFT_TENANT_ID + '/oauth2/v2.0/token?';

$http.post(urlAuth, options,
    // Callback
    function (err, response, body) {
        assert.equal(response.statusCode, 200, 'Expected a 200 OK response');

        var responseStr = response.body;
        var jsonResponse = JSON.parse(responseStr);

        var options = {
            headers: {
                'Authorization': 'Bearer ' + jsonResponse.access_token,
                'Host': 'graph.microsoft.com'
            }
        }

        $http.get('https://graph.microsoft.com/v1.0/admin/serviceAnnouncement/healthOverviews', options,
            // Callback
            function (err, response, body) {
                assert.equal(response.statusCode, 200, 'Expected a 200 OK response');

                var responseStr = response.body;
                var jsonResponse = JSON.parse(responseStr);
                //console.log('Response: ' + JSON.stringify(jsonResponse.value));   
                var jsonResponseValue = jsonResponse.value;

                jsonResponseValue.forEach((item, index) => {
                    item.eventType = NEW_RELIC_EVENT_TYPE;
                    item.location = $env.LOCATION;
                    if (item.status == "serviceDegradation") {
                        item.statusVal = 1
                    }
                    else if (item.status == "serviceOperational") {
                        item.statusVal = 0
                    }

                });
                recordData(jsonResponseValue);
            }
        )
    }
);

You can also find the latest version of the script in my GitHub repository that accompanies this blog series.

As you can see in the code, it leverages some secure credentials that are being stored in New Relic. Create each of these secure credentials in your New Relic synthetics environment. The credentials are as follows:

Secure Credential	Description
NEW_RELIC_ACCOUNT_ID	Your New Relic account ID used to store the custom events from the synthetic check
NEW_RELIC_INSIGHTS_INSERT_KEY	Your Insights insert key for your New Relic account
MSFT_TENANT_ID	Microsoft 365 tenant ID
MSFT_CLIENT_ID	Azure AD app registration client ID
MSFT_CLIENT_SECRET	Azure AD app registration client secret
MSFT_USERNAME	Azure AD username of the user on whose behalf this check is executed
MSFT_USER_PASSWORD	Azure AD password on whose behalf this check is executed

The logic of the script can roughly be broken down in three pieces:

Retrieving an access token for the user and the “virtual application” by making a request to https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token.
Leveraging the retrieved token from step 1 and making a call to Microsoft Graph API (that is, https://graph.microsoft.com/v1.0/admin/serviceAnnouncement/healthOverviews) to retrieve the service health information.
Sending the gathered information as a custom event to the New Relic platform by calling recordData(events), which in turn then posts the data with insertInsightsEvent(body).

Now that we have the health data for all of our M365 services within our subscription reported in New Relic, it’s time for us to visualize the data. The following screenshot shows a sample representation of a dashboard that can be used for getting an up-to-date overview of the service health information. The JSON representation of the dashboard that you can use to import into your New Relic account is available in my GitHub repository here.

As you can see, we can easily identify the number of services reporting some kind of degradation in the pie chart on the top left. The right-hand side shows more detail about which services are impacted.

Note: The dashboard above uses a custom visualization called Status Table Widget that’s part of the Status Widget Pack. This custom visualization is provided by New Relic and available in the Apps section under Custom Visualization in your New Relic account.

The event type for the above-mentioned data is stored in the event type that can be configured in the API script by defining the NEW_RELIC_EVENT_TYPE variable. All the raw data is available as usual in the Query Your Data part of the New Relic platform.

User login monitoring

A typical request from customers that I often hear is: “I want to be able to understand how the user experience looks for my internal users.” The information that I’m typically interested in includes the following:

Availability of the M365 websites (for example, login.microsoftonline.com).
Latency of the connection to M365 websites.
A report that shows the latency and the availability over time (for example, the last 30 days).

A pretty straightforward way to achieve this is by leveraging a New Relic synthetic scripted browser monitor.

Use the steps below to create and configure the synthetic scripted browser:

Create a new synthetic monitor and select User flow / functionality (Scripted Browser) as the monitor type.

Select your account and provide a name of the monitor; for example, Microsoft 365 - user login. Period should be set to the interval on how often you want to run the checks for the user login.

Click Select locations and select all the locations that are relevant for your business. These locations should be close to your end users who are typically using M365 on a daily business basis.
Continue to the Write script section. Here we’ll write the scripted test for the user login workflow. Copy and paste the below script content into the Script editor.
Click Save monitor.

/**
 * Feel free to explore, or check out the full documentation
 * https://docs.newrelic.com/docs/synthetics/new-relic-synthetics/scripting-monitors/writing-scripted-browsers
 * for details.
 */

var assert = require('assert');
var DefaultTimeout = 30000;

// the next function is removed for better reading purposes of the code
// full code available at https://github.com/harrykimpel/newrelic-microsoft-observability/blob/main/synthetics/user-login/user-login.js
const logger = function (...);

$webDriver.getCapabilities().then(function () { })
  // Test Case: M365  
  .then(function () {
    return Promise.resolve(true)

      .then(function () { return logger.log(1, "Open URL https://portal.office.com", "M365"), $webDriver.get("https://portal.office.com").then(e => e) })
      .then(function () { return logger.log(2, "Type username", "M365"), $browser.waitForAndFindElement($selenium.By.id("i0116"), DefaultTimeout).then(e => (e.sendKeys($secure.MSFT_USERNAME), Promise.resolve(!0))) })
      .then(function () { return logger.log(3, "Click By.id(\"idSIButton9\")", "M365"), $browser.waitForAndFindElement($selenium.By.id("idSIButton9"), DefaultTimeout).then(e => (e.click(), Promise.resolve(!0))) })
      .then(function () { return logger.log(4, "Sleep a little", "M365"), $browser.sleep (1000) })
      .then(function () { return logger.log(5, "Type password", "M365"), $browser.waitForAndFindElement($selenium.By.id("i0118"), DefaultTimeout).then(e => (e.sendKeys($secure.MSFT_USER_PASSWORD), Promise.resolve(!0))) })
      .then(function () { return logger.log(6, "Click By.id(\"idSIButton9\")", "M365"), $browser.waitForAndFindElement($selenium.By.id("idSIButton9"), DefaultTimeout).then(e => (e.click(), Promise.resolve(!0))) })
      .then(function () { return logger.log(7, "Click By.css(\"span\")", "M365"), $browser.waitForAndFindElement($selenium.By.css("span"), DefaultTimeout).then(e => (e.click(), Promise.resolve(!0))) })
      .then(function () { return logger.log(8, "Click By.id(\"idBtn_Back\")", "M365"), $browser.waitForAndFindElement($selenium.By.id("idBtn_Back"), DefaultTimeout).then(e => (e.click(), Promise.resolve(!0))) })
      .then(function () { return logger.log(9, "Click By.id(\"officeHome__content\")", "M365"), $browser.waitForAndFindElement($selenium.By.id("officeHome__content"), DefaultTimeout).then(e => (e.click(), Promise.resolve(!0))) })
      .then(function () {
        logger.endTestCase("M365");
      }, function (err) {
        logger.error(err, "M365");
        throw (err);
      });

  })

You can also find the latest version of the script in my GitHub repository that accompanies this blog series.

As you can see in the code, it leverages some secure credentials that are being stored in New Relic. Go ahead and create each of these secure credentials in your New Relic synthetics environment. The credentials are as follows:

Secure Credential	Description
MSFT_USERNAME	Azure AD username of the user on whose behalf this check is executed
MSFT_USER_PASSWORD	Azure AD password on whose behalf this check is executed

The logic of the script can roughly be broken down into three pieces:

Access https://portal.office.com.
Enter username and password for a test user from secure credentials.
Log in the user and enter the M365 portal home page.

The script will now continuously execute based on the specified period for the locations selected. If all of the steps are successful, then the status of the overall script execution will be successful. In case of any errors or timeouts, the script will report a script failure. All of the data is available in the New Relic synthetics UI.

Next steps

The two aspects mentioned above provide an easy way to get started on your journey to observe your Microsoft 365 environment. New Relic provides a free full-access account that you can use to set up service health monitoring as well as the user login monitoring scripts. Get started today!

Not an existing New Relic user? Sign up for a free account to get started!