Forem: dpacheconr

Parsing multiline logs using a custom Fluent Bit configuration

dpacheconr — Thu, 25 May 2023 17:58:01 +0000

To read this full New Relic blog, click here.

Applications generally output logs line by line, but occasionally some logs can span multiple lines to make them easier to read. While these multiline logs can improve readability when they're read consecutively (and in isolation), they can be hard to understand when queried among other logs that may appear intermixed. This becomes a problem: logs that are hard to read take up time that could be spent otherwise.

One solution is to combine all the lines from one log message into a single log record. Doing this clearly separates each multiline log, which will in turn make the log easier to understand and you save time. Although consolidating multiline log messages into a single log entry might seem daunting at first, you can do this by creating a log parser based on the common pattern within the log lines. In this post, I’ll walk you through this process. You'll learn how to use a custom Fluent Bit configuration to enable multiline log messages in New Relic logs.

If you're unfamiliar, Fluent Bit is a logging and metrics processor and forwarder. The New Relic infrastructure agent is bundled with a Fluent Bit plugin, so you can natively forward logs with the simple configuration of a YAML file.

If you’re already using Fluent Bit, you can also forward your Kubernetes logs to New Relic with the help of our Fluent Bit output plugin. Alternatively you can use it as a standalone Docker image, which we refer to as our Kubernetes plugin.

Let's get started by understanding the problem a bit better.

The challenge: When multiline logs become hard to read

Take a look at this stack trace as an example. It’s a single log made up of multiple lines. The first line starts with a timestamp, and each new line starts with the word "at".

single line...
Dec 14 06:41:08 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
at com.myproject.module.MyProject.badMethod(MyProject.java:22)
at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
at com.myproject.module.MyProject.someMethod(MyProject.java:10)
at com.myproject.module.MyProject.main(MyProject.java:6)
another line...

Now, examine how it appears in New Relic without any custom configuration applied. Notice in the next image how each line of the stack trace appears as an individual log entry.

Well that’s precisely the problem: when a single multiline log appears as multiple separate log entries, it makes the multiline log harder to read and to distinguish from other logs.

OK, let’s solve this problem together. To handle multiline logs in New Relic, let's create a custom Fluent Bit multiline parser configuration, using the same logic from the earlier examples.

Ready to follow along?

Sign up for a free account. Your account includes 100 GB/month of free data ingest, one free full-access user, and unlimited free basic users.
Get your New Relic API key.
Start instrumenting with one of our quickstart integrations.

How Fluent Bit processes and parses logs

With Fluent Bit you can gather telemetry data from various sources, apply filters to enhance it, and transmit it to any target location, such as New Relic. All log data ingested by Fluent Bit is automatically labeled, so development teams can easily apply filtering, routing, parsing, modification, and output protocols. Of all the log processors and forwarders, Fluent Bit is highly efficient and well known for being open source and vendor-neutral.

To consolidate and configure multiline logs, you’ll need to set up a Fluent Bit parser. Version 1.8 or higher of Fluent Bit offers two ways to do this: using a built-in multiline parser and using a configurable multiline parser. Together, these two multiline parsing engines are called Multiline Core, a unified functionality that handles all user corner cases for multiline logs.

In this blog post, you'll be using the configurable multiline parser from version 1.8 or higher of Fluent Bit. You'll then be able to apply the parser to a New Relic supported Kubernetes platform, the New Relic infrastructure agent, or you can use Fluent Bit as a standalone Docker image with the New Relic Kubernetes plugin.

Take a moment now to determine which version of tools you are using. Each version of New Relic uses a specific Fluent Bit version, and different versions of Fluent Bit have different features:

In Fluent Bit version 1.7 or lower, you’ll implement multiline log configuration using the old multiline configuration parameters.
In Fluent Bit versions 1.8 or higher, you’ll need to disable any old multiline configuration and instead set up a Multiline Core parser.
Any version greater than the New Relic Fluent Bit output plugin v1.12.2, Helm charts v1.10.9, or infrastructure agent v1.20.3 can use the Multiline Core functionality available in Fluent Bit v1.8 or higher.
To confirm which version of Fluent Bit you're using, check the New Relic release notes.

Creating a custom multiline parser configuration with Fluent Bit

First, it's crucial to note that Fluent Bit configs have strict indentation requirements, so copying and pasting from this blog post might lead to syntax issues. Check the Fluent Bit docs to understand the indentation requirements.

A multiline parser is defined in the parser’s configuration file by using a [MULTILINE_PARSER] section definition, which must have a unique name, a type, and other associated properties for each type.

To configure the multiline parser you must provide regular expressions (regex) to identify the start and continuation lines.

# rules   |   state name   | regex pattern                   | next state
# --------|----------------|---------------------------------------------
rule         "start_state"   "/([a-zA-Z]+ \d+ \d+\:\d+\:\d+)(.*)/"   "cont"
rule         "cont"          "/^\s+at.*/"                            "cont"

The first rule is always referred to as the “start_state”, which is a regex that captures the start of the multiline message. In the earlier example of the New Relic logs, you saw that the multiline log begins with a timestamp and continues to the end of the line.

You set the next state, “cont”, to specify how the potential continuation lines will look. You can give these custom state names and chain them together. In the example in this blog post, the continuation lines start with "at" and are targeted by a suitable regex.

You only need one continuation state for this example, but you can configure multiple continuation state definitions for more complex cases.

Make sure that you have continuation states that match all possible continuation lines. Otherwise your multiline capture might truncate when an unexpected line is encountered.

To simplify the configuration of regular expressions, you can use the Rubular website. This Rubular page shows the regex described earlier—plus a log line that matches the pattern.

Using the custom Fluent Bit multiline parser configuration

Now let’s test this out. My example uses Azure Kubernetes Service (AKS), where I deployed a New Relic Kubernetes integration using Helm. But you can apply a custom Fluent Bit configuration to any supported Kubernetes platform, or you can use it with the New Relic infrastructure agent. You can also use Fluent Bit as a standalone Docker image, which we refer to as our Kubernetes plugin.

To configure Fluent Bit within Helm, we need to make changes to the fluent-bit-config configmap to tell it to apply the parsing.

The first step is to define the correct log parser for input messages. Since I'm using the AKS cluster in this example, I need to define CRI as the log parser. Doing this ensures that each log message is first parsed using the CRI parser, before being handed over to any filters.

[INPUT]
      Name              tail
      Tag               kube.*
      Path              ${PATH}
      Parser            ${LOG_PARSER}
      DB                /var/log/flb_kube.db
      Mem_Buf_Limit     7MB
      Skip_Long_Lines   On

Parser is mapped to the value of the LOG_PARSER environment variable defined in the New Relic logging daemonset. Ensure Parser is set to “CRI” for this test, because AKS uses containerd as the container runtime and its log format is CRI-Log.

Next, within filter-kubernetes.conf add another [FILTER] section, just like in the next code snippet. This filter ensures that each log entry processed by the input log parser is then parsed using our multiline parser.

[FILTER]
    Name              multiline
    Match             *
    multiline.parser  multiline-regex

Next, within parsers.conf configure the multiline parser using the custom Fluent Bit configuration you created earlier. Notice how the multiline parser is matched by name (“multiline-regex”) from the [MULTILINE_PARSER] block in the next code snippet to the [FILTER] block in the previous code snippet.

[MULTILINE_PARSER]
    name          multiline-regex
    type          regex
    flush_timeout 1000


    # rules |   state name  | regex pattern                    | next state
    # ------|---------------|----------------------------------|-----------
    rule      "start_state"   "/(Dec \d+ \d+\:\d+\:\d+)(.*)/"    "cont"
    rule      "cont"          "/^\s+at.*/"                       "cont"

The fluent-bit-config configmap should now look like this.

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: newrelic
  labels:
    k8s-app: newrelic-logging
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     ${LOG_LEVEL}
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020
    @INCLUDE input-kubernetes.conf
    @INCLUDE output-newrelic.conf
    @INCLUDE filter-kubernetes.conf
  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              ${PATH}
        Parser            ${LOG_PARSER}
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     7MB
        Skip_Long_Lines   On
        Refresh_Interval  10
  filter-kubernetes.conf: |
    [FILTER]
        Name           multiline
        Match          *
        multiline.parser multiline-regex

    [FILTER]
        Name record_modifier
        Match *
        Record cluster_name ${CLUSTER_NAME}

    [FILTER]
        Name           kubernetes
        Match          kube.*
         Kube_URL       https://kubernetes.default.svc.cluster.local:443
        Merge_Log      Off
  output-newrelic.conf: |
    [OUTPUT]
        Name  newrelic
        Match *
        licenseKey ${LICENSE_KEY}
        endpoint ${ENDPOINT}
  parsers.conf: |
    # Relevant parsers retrieved from: https://github.com/fluent/fluent-bit/blob/master/conf/parsers.conf
    [PARSER]
        Name         docker
        Format       json
        Time_Key     time
        Time_Format  %Y-%m-%dT%H:%M:%S.%L
        Time_Keep    On

    [PARSER]
        Name cri
        Format regex
            Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z


    [MULTILINE_PARSER]
        name          multiline-regex
        key_content   message
        type          regex
        flush_timeout 1000
        #
        # Regex rules for multiline parsing
        # ---------------------------------
        #
        # configuration hints:
        #
        #  - first state always has the name: start_state
        #  - every field in the rule must be inside double quotes
        #
        # rules |   state name  | regex pattern                  | next state
        # ------|---------------|--------------------------------|-----------
        rule      "start_state"   "/(Dec \d+ \d+\:\d+\:\d+)(.*)/"  "cont"
        rule      "cont"          "/^\s+at.*/"                     "cont"

As the final step, you'll need to restart the New Relic logging pods after changing the configuration.

Confirming the custom multiline parser is working correctly

Now within New Relic you can see that the log messages are no longer split across multiple lines and the full stack trace can be seen as part of a single log line.

A seemingly small change like this can have a big impact when you're dealing with multiline logs, and it ultimately makes your debugging experience with logs more effective. Thankfully, this small change is no big deal to implement.

Next steps

If you aren’t already using Fluent Bit, start by enabling log forwarding with the Fluent Bit plugin. Then, you’ll be ready to set up a custom configuration for parsing your multiline logs.

If you're not using New Relic logs yet, sign up for a free account. Your account includes 100 GB/month of free data ingest, one free full-platform user, and unlimited free basic users.

To read this full New Relic blog, click here.

Monitor GitLab with OpenTelemetry and New Relic

dpacheconr — Mon, 06 Mar 2023 18:33:43 +0000

Shifting left isn't just about testing your code and automating the build and deployment process. It's about monitoring and testing your CI and CD pipelines, too.

In this post, you’ll learn how to monitor your GitLab jobs and pipelines using OpenTelemetry and the New Relic GitLab quickstart. After you're finished, you'll be able to:

Track key metrics on your GitLab pipelines in a pre-built dashboard, such as how long your jobs are taking and how often they are failing.
Visualize jobs and pipeline executions as distributed traces with logs in context.
Pinpoint where issues are coming from in your pipelines.
Create alerts on your GitLab pipelines.

The next image shows a New Relic dashboard with some of the GitLab metrics you’ll be able to see.

With this integration, you can shift left and get observability for the build and deploy phases of the DevOps lifecycle.

How the integration works

The New Relic GitLab integration is split into two parts for ease of use. Both parts run as jobs in your existing or new GitLab pipelines. You’ll learn how to integrate each part in the next section.

New Relic exporter: Add this job to the end of each of your pipelines to export your pipeline data as logs and traces to New Relic. You’ll be able to use the distributed trace viewer to inspect each pipeline run as well as explorer logs in the log explorer.
New Relic metrics exporter: This job should be scheduled. It exports GitLab data for all your projects, pipelines, jobs, runners, environments, deployment, and releases. You can configure which groups and projects are included. You can also run this as a standalone container outside GitLab.

Let’s look at some examples that show what you can visualize in New Relic after you’ve enabled the integration.

The following distributed trace captured by the New Relic exporter shows a job that contains an error with associated logs in context.

The next two images show metrics captured by the New Relic metrics exporter.

The next image shows GitLab resources as log events with the configured attributes.

Configuring the exporters

Both exporter integrations are set up similarly, but with one key difference:

The New Relic exporter needs to be set up for each pipeline you want to report data for.
The New Relic metrics exporter only needs to be set up once.

Prerequisites:

Before setting up the integration, you will need both a GitLab token and a New Relic ingest API key.

Follow these steps in the API documentation to generate a New Relic ingest API key.
Generate a GitLab API Token with “read_api” access.

In the top-right corner, select your avatar.
Select Edit profile.
On the left sidebar, select Access Tokens.
Enter a name and optional expiration date for the token.
Select the “read_api” scope.
Select Create personal access token.

Setting up the New Relic exporter

The New Relic exporter should be configured to run as a child job at the end of your project pipeline. It will ship the data from the pipeline to New Relic as a distributed trace with the logs being reported as logs in context.

Step one: Create CI/CD variables in GitLab

First, you need to set up your CI/CD variables in GitLab. In addition to the required GitLab token and New Relic API key, there are also several optional variables.

To add or update variables in GitLab for your project:

Go to your project’s Settings > CI/CD and expand the Variables section.
Select Add variable and fill in the details for the following required variables. Make sure that both variables are masked. GLAB_TOKEN provides access to the GitLab API. Add the GitLab API token you created in the previous section. NEW_RELIC_API_KEY is the ingest API key you created in the previous section.
You can also update the following optional variables. Each configuration option should be set up in GitLab as a pipeline environment variable. By default, these optional variables are already set to automatically export job logs to New Relic. 1.OTEL_EXPORTER_OTEL_ENDPOINT is the New Relic OTEL endpoint and includes a port. By default, it’s set to https://otlp.nr-data.net:4318 in the US and https://otlp.eu01.nr-data.net:4318 in the EU. 2.GLAB_EXPORT_LOGS is used to export job logs to New Relic. This variable is a boolean that’s set to true by default.
GLAB_ENDPOINT is the GitLab API endpoint. It’s set to https://gitlab.com by default.

Step two: Add the pipeline configuration for the exporter

Create a file called new-relic-exporter.yml in the root directory of the GitLab project, repository, or branch you want to monitor.

Add the following pipeline code. The New Relic image is provided for testing purposes. It’s recommended that you use your own image of the exporter in production. You can build your own image based on the repository.

stages:
- new-relic-exporter
new-relic-exporter:
 rules:
   - when: always
 image:
   name: docker.io/dpacheconr/gitlab-exporter:1.0.0
 stage: new-relic-exporter
 script:
   - echo "Done"

Ensure that the rule is set to always so that the exporter runs even if previous jobs fail.

Step three: Add the exporter job to your existing pipeline

In your .gitlab-ci.yml file, add a new stage with the name new-relic-exporter to your stages block as shown in the next code snippet. This should be set as the last stage in the list.

stages:
…existing stages…
- new-relic-exporter

Configure the new-relic-exporter stage block as shown in the next code snippet:


new-relic-exporter:
 stage: new-relic-exporter
 inherit:
   variables: true
 variables:
   CI_PARENT_PIPELINE: $CI_PIPELINE_ID
 trigger:
   include:
     - local: new-relic-exporter.yml

Step four: Test and confirm that New Relic is receiving data

Commit your code to run your pipeline and confirm that the New Relic Exporter job runs as the final job in the pipeline.

Log into New Relic, go to Services - OpenTelemetry, and select the entity. It will have the same name as the project, repo, or branch that you are monitoring. Select the Distributed Traces tab and confirm that you can see a trace for your pipeline. Sometimes it takes a few minutes for the first trace to appear. The next image shows an example of a trace from the exporter.

An example of a trace from the exporter in the distributed traces dashboard in New Relic.

Next, go to the Logs tab and confirm that you can see your pipeline logs appearing.

Log events in New Relic show GitLab resources.

If there is no data, ensure that you have correctly set up the environment variables with the appropriate keys. In GitLab, you can check the New Relic exporter job output for additional debugging information.

Setting up the New Relic metrics exporter

The New Relic metrics exporter is standalone and does not need to be attached to an existing project pipeline. You should run it in its own project pipeline, but you can add it to an existing pipeline if you prefer.

This gathers data from the GitLab API about your projects, pipelines, jobs, runners, environments, deployments, and releases, then exports the data to New Relic as a mixture of metrics and logs.

To get the latest data from your accounts, you should run the New Relic metrics exporter on a regular basis. At least once per hour is recommended.

Step one: Create a new project

Create a new GitLab project and give it a descriptive name like New Relic metrics exporter.

Step two: Create CI/CD variables in GitLab

The exporter job has a number of configurable options which are provided as CI/CD environment variables. You must supply a GitLab token and New Relic API key.

You also need to provide a regular expression to define which projects and groups you’d like the exporter to monitor. Your GitLab token may have access to data you don’t want to export so take care to set this up so that only the projects and groups you are interested in are captured.

The other configurations are optional. Each configuration option should be set up as a pipeline environment variable. See the README for a list of the configuration options available along with their default values. The defaults with no additional configuration will run the job every 60 minutes.

To add or update variables in project settings:

Go to your project’s Settings > CI/CD and expand the Variables section.
Select Add variable and add your GLAB_TOKEN and NEW_RELIC_API_KEY. Don’t forget to mask both variables.
Add or update any optional variables as needed for your project. If you want to change the frequency of the job to a value other than 60 minutes, you can do so with the optional variable GLAB_EXPORT_LAST_MINUTES.

Step three: Create your pipeline definition

If you have created a new project for this exporter as recommended, you will need to create the pipeline definition file .gitlab-ci.yml in the root of your project. Add the pipeline code shown in the next snippet.

stages:
- new-relic-metrics-exporter
 new-relic-metrics-exporter:
 rules:
   - when: always
 image:
   name: docker.io/dpacheconr/gitlab-metrics-exporter:1.0.0
   pull_policy: always
 stage: new-relic-metrics-exporter
 script:
   - echo "Done"

Step four: Test the pipeline

Run the pipeline and confirm that the New Relic metrics exporter job runs correctly with no errors, by checking the New Relic metrics exporter job output.

To execute a pipeline schedule manually, follow these steps in GitLab:

On the top bar, select Main menu > Projects and find your project.
On the left sidebar, select CI/CD > Schedules.
Select Play to run the desired pipeline.

In the New Relic query builder, run the following NRQL query to ensure that New Relic is receiving data from the exporter:

SELECT * FROM Log where gitlab.source ='gitlab-metrics-exporter'

Go to Metrics & Events in New Relic and your GitLab metrics in the list of available metrics. These metrics will be prefixed with gitlab.

Metrics include pipeline, job duration, and queue duration. The next image shows the information on the duration of GitLab jobs.

New Relic dashboard shows duration of GitLab jobs.

Go to Logs in New Relic to see your log events. Apply the filter gitlab.source:"gitlab-metrics-exporter" to only see data sourced from new-relic-metrics-exporter. You will be able to see log data on all of your GitLab resources, including your projects, deployments, and runners.

All log data showing in New Relic, including GitLab projects, deployments, and runners.

Step five: Create a pipeline schedule

Once you have confirmed the job is correctly sending data, you need to set up a schedule in GitLab to run the job regularly. The schedule pattern should match the value from GLAB_EXPORT_LAST_MINUTES. As discussed earlier in this post, the default is 60 minutes. Setting values that are different from the value in GLAB_EXPORT_LAST_MINUTES can lead to duplicate or missing data in New Relic.

To add a pipeline schedule, do the following in GitLab:

On the top bar, select Main menu > Projects and find your project.

On the left sidebar, select CI/CD > Schedules.

Select New schedule and add an interval pattern To run hourly, set the pattern to  0 * * * *.

Step six: Import the quickstart dashboard

We’ve built a starter dashboard to visualize your GitLab data and help you monitor your jobs and pipelines. You can install this dashboard using the GitLab quickstart and then further customize it as needed. To get this dashboard, simply follow these steps:

Go to the Gitlab quickstart in New Relic Instant Observability and select Install now.
Choose an account and select Begin installation.
If you've already completed the earlier steps in this tutorial, select Done to move on to the next step.
The quickstart deploys the resources to your account. Select See your data to get to the dashboard.

The following image shows the dashboard, which provides a summary of key metrics such as the number of runners available, the number of successful jobs, the top errors, and the job queue status.

Install the GitLab quickstart to get a dashboard template with prebuilt views to monitor your GitLab pipelines.

Conclusion

After you're done, you'll have complete visibility into your GitLab estate, from high-level metrics to logs and distributed traces. The next step is to consider adding alerts when certain thresholds are met, such as when jobs are taking too long, or when the number pipeline failures gets too high.

By sending your GitLab data to New Relic, you get observability into the building and deployment phases of the DevOps lifecycle, providing you with more insights into your DevOps practices and helping you shift left.

Read the full blog post at New Relic: Monitor GitLab with OpenTelemetry and New Relic.

Not an existing New Relic user? Sign up for a free account to get started! 👨‍💻