Forem: Liz Fong-Jones

Making Instrumentation Extensible

Liz Fong-Jones — Wed, 09 Dec 2020 20:06:08 +0000

Observability-driven development requires both rich query capabilities and sufficient instrumentation in order to capture the nuances of developers' intention and useful dimensions of cardinality. When our systems are running in containers, we need an equivalent to our local debugging tools that is as easy to use as Printf and as powerful as gdb. We should empower developers to write instrumentation by ensuring that it's easy to add context to our data, and requires little maintenance work to add or replace telemetry providers after the fact. Instead of thinking about individual counters or log lines in isolation, we need to consider how the telemetry we might want to transmit fits into a wider whole.

Aggregated counters, gauges, and histograms can provide us with information broken down by host or endpoint, but not necessarily higher cardinality fields that we need for a rich understanding of our distributed systems. Automatic instrumentation of a language server framework, such as framework support for Node.js Express or Go http.Server in Honeycomb's Beelines, can only provide a modest amount of context. It will capture request header fields such as URL and response durations/error codes, but not anything from the business logic or involving the logged-in user's metadata. Because observability requires the ability to understand the impact of user behavior upon our applications, we cannot just stop with collecting surface level data. Thus, we'll need to make changes to our code to instrument it.

Instrumentation should be reusable

Typically, instrumenting code involves adding a vendor's library or a standard package like OpenCensus or slf4j to one's dependencies, then calling the library directly from instrumented code. If multiple providers and kinds of telemetry (e.g. logs, metrics, traces, events…) are in use, calls to each wind up sprinkled across the codebase. But should we have to re-instrument our entire codebase every time we gain access to new methods of data aggregation/visualization or change observability providers? Of course not. This gives rise to the need to separate observability plumbing from your business logic, or domain-specific code.

To address this problem of abstracting instrumentation, Tyler Treat envisions a solution involving centralized inter-process collection in "The Observability Pipeline", and Pete Hodgson suggests abstracting collection in-process in "Domain Oriented Observability". Tyler's article explains creating structured events, and then streaming them to an out of process service that can aggregate them and send them onwards to a variety of instrumentation sinks. Pete's article suggests creating a separate class for handling the vendor-specific pieces of instrumentation, but still relies upon tight coupling between the instrumentation code and the domain-specific code--e.g. creating a method in the instrumentation code to handle each potential property we might want to record about an event (e.g. discountCodeApplied(), discountLookup{Failed,Succeeded}()).

Why not both?

However, there's a simpler, within-process approach that is easier for developers to understand, test, configure, maintain, and operate. It's a fusion of that described by Pete in "Event-Based Observability" and "Collecting Instrumentation Context", and by Tyler's distributed event buffering solution. With the improved solution, we neither need to have an advanced understanding of mocking functions and classes, nor do we need to operate a Kafka pipeline from day 0. Instead, we just generate and consume structured events.

Within each span of work done in the domain-specific business logic, we populate a weakly-typed context dictionary with key/value pairs added from within instrumented code, as well as the default standard contextual fields (e.g. requestId, startTime, endTime, etc). Child units of work become separate contexts and spans, with appropriate fields (e.g. parentId, requestId) templated (or "partially applied" in Pete's words) from the parent context/span. Adding telemetry becomes as easy as Printf for developers -- it's just setting a ctx[key] = val only for keys and values relevant to your code. We no longer need to create one function call to the instrumentation adapter for each telemetry action. Using Pete's example, we might set discountCode => FREESHIPPING, responseCode => 403, or discountLookupSucceeded => {true,false,nil} within one event instead of making the multiple function calls above, or emitting multiple distinct "Announcement" objects for only one work unit. Writing tests to validate that the generated context map is correct becomes straightforward to do in table-based testsuites (e.g. go functest), rather than requiring mocking functions and classes.

Once the work unit finishes, its context dictionary is sent in-process to the instrumentation adapter where any number of listeners can interpret it. Each listener sees the context maps for each received event, decides whether it's relevant to it, and if so, translates it according to its own rules into metrics, traces/events/structured logs, or human-readable logs. We no longer need to duplicate calls to the same instrumentation provider from each kind of telemetry function call, but can create single listeners for each common metric (e.g. response time metrics collection, response code) that act on a wider range of events. We can then measure the correctness of listeners, ensuring that each processor is only interested in the correct set of structured events, and dispatches them to the upstream structured event, log, metric, or trace provider(s)' APIs appropriately.

Correspondence is more useful when it's about the outcome

Unlike Tyler's streaming design there need not be a 1:1 correspondence between listeners/routers and instrumentation sinks. Instead, the correspondence is between the action we'd like to coalesce or report on, and what related calls we make -- e.g. performing more than one metric counter increment, etc. to the same sink, or even scattering increments across many different sinks if we're transitioning between providers. This makes the code much more testable, as it's focused on the intent of "record these values from this specific kind of event, to whatever Sinks are relevant", rather than a catch-all of "duplicate everything in the kitchen we do in Sink A in Sink B instead". And the value of event stores such as Honeycomb quickly becomes clearer -- because you don't have to do anything different to aggregate or process each such structured event, only pass it on to us directly. Let us worry about how to efficiently query the data when you ask a question, such as P99(duration_ms) or COUNT WHERE err exists GROUP BY customer_id ORDER BY COUNT desc.

Decoupling event creation from event consumption, even within the same process, is a great step between instrumentation spaghetti and needing a Kafka, Kinesis or PubSub queue. Never create a distributed system unless you need to, and run as few distributed systems as possible. Same-process structured event creation and structured event consumption is super easy to work with, test, and reason about, to boot! As you grow and your needs scale, you may wind up reaching for that Kafka queue. But you'll have an easier migration path, if so.

Ideas for future-proofing

How does this relate to OpenTelemetry née Open{Census,Tracing}? Despite the creation of the new consensus standard, the ongoing transition to OpenTelemetry is proof indeed that we ought to future-proof our work by ensuring we can switch to and from instrumentation providers, including those that do not support the current newest standard, without further breaking domain code. Instead of using the OpenTelemetry API directly within your domain-specific code, it still may be wise to use one context/span propagation library of your choice (which could still be OTel's), and write an InstrumentationAdapter that passes data it receives through to OpenTelemetry's metrics & trace consumers, as well as to legacy and future instrumentation providers.

I hope that this article was helpful! If you're looking for more detailed examples of how Honeycomb Beelines work, check out our Examples repo in Github , such as this example of using our Beeline for Go alongside custom instrumentation.

Looking to find out more? Get started with Honeycomb for free.

From 0 to Insight with OpenTelemetry in Go

Liz Fong-Jones — Mon, 09 Nov 2020 19:26:56 +0000

Honeycomb allows you to send in data from a variety of sources. The steps to get there are a choose-your-own-adventure path and the choices to make may not always be clear to folks new to Honeycomb. This guide is aimed at developers that own Go applications in production and would like to send instrumentation data to Honeycomb using OpenTelemetry.

This step-by-step guide will help you add OpenTelemetry to your web service, leverage automatic instrumentation, show you how to add additional custom context to that instrumentation, and ensure that instrumentation data is being sent to Honeycomb. This is a zero-to-insight follow along guide to getting started quickly.

Create a free Honeycomb account

There are several ways to get started with Honeycomb. For simplicity, we’ll start by signing up for an account in Honeycomb’s free tier. Go to ui.honeycomb.io/signup to create a free honeycomb account. We support SSO with Google and if you choose that, there are no new passwords to create or remember!

Provide your organization’s name when prompted. This is a unique ID, so you can either use your company's name, or if it's a large company you might go with company-team, e.g. huuli-research.

If you're prompted to name your first dataset, give it the same name as your app. (If you change your mind about the name later, you can just create a new dataset.) Select the Go integration and click Create.

Once you’ve created your team, you’ll automatically receive an email inviting you to the Honeycomb Pollinators Slack community. This blog post from Customer Advocate Jenni Boyer details why you should join, definitely check it out when you get the chance!

At this point you may be prompted to choose how you want to send data to Honeycomb. Click “Show All Integrations”, then “OpenTelemetry”. Now you’ll see a prompt with your API key and some sample code. But instead of using the sample code shown to you in the start up instructions, follow along here with the rest of this post, as these instructions here are more comprehensive and detailed.

Grab your API key and let's continue on to set up the configuration in your Go app.

Automatically create traces/spans on HTTP requests

OpenTelemetry can help you jumpstart your way to observability by providing automatic instrumentation for HTTP requests. You have your choice of request routers in OpenTelemetry or you can use the standard HTTP handler. You should pick the mux that’s right for your framework.

Automatic instrumentation with request routers

Add one line to your import() stanza depending upon your request router:

If you are using gin/gonic:

middleware "go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin"

If you are using gorillamux:

middleware "go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux"

If you are using echo:

middleware "go.opentelemetry.io/contrib/instrumentation/github.com/labstack/echo"

Inject OpenTelemetry middleware

Then, in your main() function, right after you create your primary router, inject the OpenTelemetry middleware:

router.Use(middleware.Middleware("service-name"))

Yes, it was that simple with request routers. Two lines of code!

If you don’t use a request router

import (
  otelhttp "go.opentelemetry.io/contrib/instrumentation/net/http"
)

In each place where you pass an http.Handler to a ServeMux, you’ll wrap the handler function. For instance, you'll make the following replacements:

- mux.Handle("/path", h)
+ mux.Handle("/path", otelhttp.NewHandler(h, "description of path"))

- mux.Handle("/path", http.HandlerFunc(f))
+ mux.Handle("/path", otelhttp.NewHandler(http.HandlerFunc(f), "description of path"))

In this fashion, you can ensure that every function you wrap with othttp will automatically have its metadata collected and a corresponding trace started.

Automatically create traces/spans on gRPC server requests

Similarly, OpenTelemetry can also help you automatically instrument gRPC requests. To instrument any gRPC servers you have, add anInterceptor to the instantiation of the server.

import (
  grpcotel "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc"
)

func main() {
  [...]

  s := grpc.NewServer(
    grpc.UnaryInterceptor(grpcotel.UnaryServerInterceptor(global.Tracer(""))),
    grpc.StreamInterceptor(grpcotel.StreamServerInterceptor(global.Tracer(""))),
  )
}

Configure export to Honeycomb

Now that your code is set up to emit instrumentation, you’ll need to send it to Honeycomb in order for it to be useful. We’ve written a Honeycomb exporter for OpenTelemetry to help make that easier. You’ll need to import the Honeycomb exporter in your main module, and place the initialization code first in your main() function. Use the API key you found in the getting started screen from the Honeycomb website...

import "github.com/honeycombio/opentelemetry-exporter-go/honeycomb"
import "go.opentelemetry.io/otel/api/global"

func initTracer() func() {
  apikey, _ := os.LookupEnv("HNY_KEY")
  dataset, _ := os.LookupEnv("HNY_DATASET")
  hny, err := honeycomb.NewExporter(
    honeycomb.Config{
      APIKey: "*0123456789abcdef0123456789abcdef*",
    },
    honeycomb.TargetingDataset(dataset),
    honeycomb.WithServiceName(serviceName),
  )
  if err != nil {
    log.Fatal(err)
  }
  tp, err := sdktrace.NewProvider(
    sdktrace.WithConfig(sdktrace.Config{DefaultSampler: sdktrace.AlwaysSample()}),
    sdktrace.WithSyncer(hny),
  )
  if err != nil {
    log.Fatal(err)
  }
  global.SetTraceProvider(tp)
  return hny.Close
}

func main() {
  cleanup := initTracer()
  defer cleanup()
  // rest of initialization, including creating HTTP and gRPC servers/handlers...
}

Add custom instrumentation

OpenTelemetry includes a fair amount of automatic instrumentation that can help you get started quickly. However, you’ll likely soon discover that it would be helpful to understand additional bits of context for each event that is now emitting automatic instrumentation data. This is where the power of observability particularly shines. Honeycomb encourages you to send in as much context as you need for these events. You can make your events incredibly wide with no penalties for doing so. I know we’re using the Free tier for this tutorial, but even in that tier the only limitation is 20M events per month. Each of those events can include as much additional context as you want to add in: it still only counts as one event to Honeycomb.

(Yes, there’s actually a cap on max event size, but for most folks this is essentially unlimited).

Attributes

Attributes are any arbitrary key value pairs you can add to any event. Anything at all! If there’s a property you want to see that isn’t captured by the HTTP instrumentation, that’s not a problem! Inside of any HTTP handler, you can access the current span from the request’s Context() and add any number of key-value pairs using SetAttributes().

import "go.opentelemetry.io/otel/api/kv"
import "go.opentelemetry.io/otel/api/trace"

func h(w http.ResponseWriter, req *http.Request) {
  ctx := req.Context()
  span := trace.SpanFromContext(ctx)
  span.SetAttributes(kv.Any("id", id), kv.Any("price", price))
  // [...]
}

Child spans

Sometimes you want to track units of work performed within a single HTTP request. For instance, suppose you needed to look up 50 different items inside a function or goroutine before returning the results to the user. OpenTelemetry allows you to easily create a new span with tracer.Start().

For instance, in your cart/item_lookup.go file for a hypothetical shopping service:

import "go.opentelemetry.io/otel/api/global"
var tracer = global.Tracer("cart")

func getItem(ctx context.Context) {
  _, span := tracer.Start(ctx, "item")
  defer span.End()
  // [...]
}

This creates a child span for each item that is retrieved, allowing you to understand whether the items are being fetched in serial or parallel, and how long each item is taking to retrieve.

Context propagation

Part of why tracing is so useful is because it can be distributed across multiple services. In order to get the most benefit out of distributed tracing, you’ll need to make sure that any downstream services that are issued HTTP client calls, or gRPC client calls, are aware of your traces so that you can build upon them. You’ll need to add OpenTelemetry to those services to see their spans once you’ve added propagation between services.

For HTTP calls, you’ll need to specify a custom http.Client instrumented with OpenTelemetry hooks. Instead of:

func remoteCall(...) ... {
  http.Get(url)
}

You’ll write:

import (
  otelhttp "go.opentelemetry.io/contrib/instrumentation/net/http"
)

var httpClient = http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}

func remoteCall(...) ... {
  req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
  resp, err := httpClient.Do(req)
}

For gRPC, you’ll just add an Interceptor to your outgoing connections, without needing to modify your gRPC calls:

import (
  grpcotel "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc"
)

func main() {
  var conn *grpc.ClientConn
  conn, err := grpc.Dial("remotehost:1234",
    grpc.WithUnaryInterceptor(grpcotel.UnaryClientInterceptor(global.Tracer(""))),
    grpc.WithStreamInterceptor(grpcotel.StreamClientInterceptor(global.Tracer(""))),
  )
}

Start observing your application!

Now that you’ve added OpenTelemetry and custom instrumentation to your code, you’ll need to deploy and run it in order for any data emitted to the Honeycomb exporter to send it to Honeycomb. Once you deploy your binary (or multiple binaries) with the changes you just made above, the “waiting for data...” screen in the Honeycomb UI will let you proceed into the analysis homepage, where you can see a summary of your data.

You can see the list of recent traces at the bottom of the home view, and can inspect a trace to verify it shows the spans and fields you instrumented. You can start digging around for more sophisticated insights using the ‘New Query’ button to create and issue custom queries against that data.

Congratulations! You’ve just added your Go application to Honeycomb using OpenTelemetry. Have questions or comments? Go to your inbox and accept that invite to join the Honeycomb Pollinators Slack group. We look forward to seeing you there!

Now that you've gotten data in, learn how to reach the next level of your observability practice: download the Observability for Developers whitepaper.

First experiences with Honeycomb LX2K

Liz Fong-Jones — Wed, 13 May 2020 15:21:09 +0000

I last wrote about arm64 two months ago when I was excited for my Honeycomb LX2K board.

Well, it's arrived! and it's glorious! Everything is just working, after some config tweaks and patching. I can stay connected with my team via Slack, write code, and deploy to Graviton2 in the cloud without touching an amd64 machine.

Here are the photos of the board and its installed setup inside the chassis!

Here's the missing manual, written on the very same Honeycomb LX2K arm64 board I did this setup on!

Initial setup

Set up the hardware
- Insert the NVMe boot drive
- Insert RAM sticks
- Plug in the PicoPSU
- Plug in the GPU
- Attach a low-noise fan adapter inline with the CPU fan to avoid the jet engine sounds (PWM isn't working yet).
- Attach the case button(s) to the power and reset jumpers; reset is more important than power.
- Do not yet attach a monitor or mouse/keyboard.
Flash the latest UEFI firmware to a micro SD card
- Untar the image folder
- dd if=lx2160acex7_2200_700_2400_8_5_2.img of=/dev/sda where sda is the SD card (make sure it's not mounted, and really is your SD card and not some other load-bearing device)
- Plug the SD card into the board. Make sure the firmware source jumpers are set off-ON-ON-ON-ON-off (or otherwise as indicated by silkscreen on board).
Boot for the first time.
- Plug an Ubuntu ARM64 20.04 USB stick into the board (created with Startup Disk Creator & the latest 20.04 ARM image)
- Plug in an ethernet cable via USB adapter (the ports on the board won't work yet)
- Plug in the micro-USB serial console port to an existing machine, and run minicom configured to talk to /dev/ttyUSB0 with hardware/software buffering OFF (very important!)
- Power on the device by inserting the power cable or turning on the PSU
Install Ubuntu 20.04
- If Ubuntu installer GRUB menu doesn't show, hit esc during boot and manually select the USB stick as the boot device.
- At GRUB menu, select install rather than try live
- You'll need to update the installer snap when prompted, as the base ISO doesn't install on ARM64 properly
- Make sure you enable SSH, so you can do the next bit of work comfortably from a desktop environment
- Disk encryption just works.
- Reboot when prompted; if it doesn't cleanly shut down, kick the reset button after the prompt to remove the USB stick.
Boot the second time.
- Enter the FDE password over serial console
- Once system is booted, ssh to it from your desktop to make sure everything works.

Prepare to enable graphical mode

Configure AMD PCIe settings: set GRUB_CMDLINE_LINUX_DEFAULT="amdgpu.pcie_gen_cap=0x00040004" in /etc/default/grub then run sudo update-grub
Install the PCIe bug patched kernel image PPA: https://launchpad.net/~lizthegrey/+archive/ubuntu/kernel and install the linux-image-unsigned-5.4.0-42-generic package.
Now, reboot, with display and mouse/keyboard attached. If it doesn't work, you can disconnect display and fall back to serial console to debug (and ask for help in SolidRun Discord)

Enabling graphical mode

Log in after rebooting (verifying the new kernel is loaded)
sudo apt install ubuntu-desktop
mkdir -p ~/.config/environment.d && echo 'export MESA_EXTENSION_OVERRIDE=-GL_ARB_buffer_storage' > ~/.config/environment.d/00-mesa.conf
If needed, run sudo service gdm3 restart and/or sudo systemctl isolate graphical.target. GDM should be showing. If not, reboot.
You will want to log in with wayland, not with the default Xorg; click the gear icon in the lower right hand corner to use Ubuntu with Wayland after clicking your name, before entering your password.
Congratulations, you have a working Gnome3 desktop.

Helpful things to install

First off, you probably want a web browser. sudo apt install chromium-browser will give you a chromium updated with snap, supported by Ubuntu, for arm64, if you don't like Firefox.
- sudo update-alternatives --install /usr/bin/x-www-browser x-www-browser /snap/bin/chromium 200 will make it your default.
- Yes, security keys work out of the box!
- You can get graphics acceleration by running snap refresh chromium --channel=candidate/vaapi
Everything else should just work to taste when running apt install -- e.g. golang-1.14, openjdk, etc.

Year of ARM64 on the desktop

Liz Fong-Jones — Mon, 09 Mar 2020 17:14:04 +0000

After getting my first exposure to aarch64/ARMv8/ARM64 at work the past few weeks, I'm curious to not just use it in the cloud, but also under my desk! We are cross-compiling from AMD64 to create ARM64 binaries for use in a subset of our canary environment, but it's started feeling weird to me that I cannot locally test my changes before they go out to production.

So, yesterday I ordered a Honeycomb LX2K Mini-ITX board, no relation to my employer honeycomb.io, and I'll have it up and running ~~sometime next week~~in May due to lack of stock! This post will be updated with my build and findings! and I've written up the install instructions

Part list (with the help of /r/sffpc):

Lone Industries L5 case
Minibox 160 XT PSU
Radeon Pro WX 3200 half-height GPU
Honeycomb LX2K Mini-ITX board
~~Any old DDR4 RAM~~ G.SKILL Ripjaws Series 16GB (2 x 8GB) 260-Pin DDR4 SO-DIMM
512GB M.2 SSD

I'm excited to try:

Native builds of Debian/Ubuntu packages, with my own GPG keys available
Wine Hangover for x86_64 games
Seeing if I can get Zoom, Chrome, etc. to work given they're built for ChromeOS aarch64 which is Linux under the hood