Forem: shelby spees (she/her)

How to set up Twitter cards for your Hugo site

shelby spees (she/her) — Mon, 14 Dec 2020 05:59:21 +0000

My Hugo site is basically for vanity. So of course when I linked to one of my blog posts on Twitter today and the unfurl was super ugly, I had to go and fix it.

Grody.

So how to fix? First we need to connect the dots on how this unfurl happens.

What are social cards?

An occupational hazard of doing devrel and technical marketing is that you learn that companies do a lot to optimize how their links look on social media.

For any particular image post, you need to create multiple sizes so the image shows up correctly on each platform: Twitter vs. Facebook vs. LinkedIn vs. Instagram, etc. The best aspect ratio for Instagram is a square, but these other platforms couldn't agree on a standard aspect ratio or image size so the burden is on you to adjust for each one. Thankfully, there are helpful articles from SEO people that list all of the varying sizes.

Size is important for image quality, sure, but you can't just throw in a giant .png and call it a day. Aspect ratio is critical when the image contains text that you don't want to get cut off, like event details. Brands also care about less critical things like spacing and alignment, because bad visual flow can reflect poorly on the company. Priorities differ--image margins are less important when I'm Joe Schmoe Developer and more important when I'm Microsoft or Coca-Cola, but it's good to be armed with such knowledge.

If you think making different image sizes for each platform sounds like a pain in the ass, you'd be correct. Thankfully, tools like Canva and Adobe Spark make this easier. Visual designers can create templates that marketers can fill in with content and export in the various sizes. Developers can attach the exported image assets to the original source website, or marketers can manually add the images to social posts on each platform. My company creates social posts the second way, uploading the image to Hubspot because we often want to customize the image and post content separate from the link. What I'm trying to do for my blog is the first way--having an image live in my website's files so that social platforms (specifically Twitter, in this case) can grab it for the unfurl every time my link is shared.

Twitter cards

So for any link dropped into a tweet, Twitter will look at content of that page and try to grab relevant information to display in the unfurl--called a Twitter card. Twitter also has a handy Card Validator tool for checking how your link unfurl will look before you hit "publish" on a post. Here's what it looks like for Google.com:

The validator also gives you a handy log from the JavaScript that's grabbing the page metadata:

INFO:  Page fetched successfully
INFO:  17 metatags were found
INFO:  twitter:card = summary_large_image tag found
INFO:  Card loaded successfully
WARN:  this card is redirected to https://www.google.com/

Looking at the preview for my website:

I get mostly the same info, although without the redirect warning:

INFO:  Page fetched successfully
INFO:  15 metatags were found
INFO:  twitter:card = summary_large_image tag found
INFO:  Card loaded successfully

So where is it getting this twitter:card = summary_large_image tag it says that it found? The previous line gives us a hint: metatags. Let's open up Chrome devtools.

I highlighted these four meta tags because they correspond to the info displayed in the Twitter card preview:

<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:image" content="https://shelbyspees.com/nova-ears.png">
<meta name="twitter:title" content="shelby spees">
<meta name="twitter:description" content="shelby spees: developer advocate, dog mom, etc.">

The first one says summary_large_image and it seems to be pointing to nova-ears.png, which is the image on my website's my home page. But like, where did all this get defined? I don't remember specifying anything for Twitter.

Hugo Twitter cards

Oh wait nvm, apparently I thought I knew what I was doing.

Well sometime between September 22nd and December 8th, I forgot that I had done that. I guess at the time I got frustrated and gave up? Then last week I made a post about being on Screaming in the Cloud and was reminded by my RSS feed tweeting about it:

That's when I finally bothered to look into it (evidently not for the first time) and (re)learned: Hugo supports Twitter cards out of the box!

The docs say that the _internal/twitter_cards.html template grabs the image path from my site's config file, in the images list under params, which is what I set in the commit I shared above:

[params]
  defaultTheme = "light"
  homeSubtitle = "developer advocate, dog mom, etc."
  mainSections = ["posts"]
  showReadMore = true
  gitUrl = "https://github.com/shelbyspees/website/commit/"
  description = "shelby spees: developer advocate, dog mom, etc."
  images = ["/nova-ears.png"] <-- this one

Since nova-ears.png is square, it ends up looking super wonky in the 2:1 ratio of the Twitter card. I'm going to have to replace it.

Hugo layouts

To be honest though, I still felt the need to connect the dots a bit more. Where in my theme is this getting used? (Note: I keep my theme as a git submodule separate from the main content of my website.)

Thankfully I know Hugo enough to start in my theme's layouts/ folder. You would think to go to layouts/index.html, but Hugo templating means things are split up a bit more. There's a layer of template beyond that, under layouts/_default/:

$ ls -1 layouts/_default
baseof.html
list.html
single.html

The one we want is layouts/_default/baseof.html, which we can confirm because it starts with <!DOCTYPE html> and goes on to include the head tags:

<!DOCTYPE html>
<html lang="{{ .Site.Language }}">
    <head>
        {{ partial "head.html" . }}
    </head>
    ...

(To be honest, it's not immediately clear from the content of layouts/index.html that it's using layouts/_default/baseof.html under the hood. I've learned a lot of Hugo templating logic just by poking around and adding testeststests and Shelby was here in random spots to see where it showed up.)

That partial "head.html" bit is our next clue. Hugo, like other templating engines, supports the use of partial templates, which are like modular mini-templates that you can import into other templates. Looking at layouts/partials/head.html, there it is on line 40:

{{ template "_internal/twitter_cards.html" . }}

Awesome! The theme developer put the internal Hugo Twitter card template there so I don't have to! That explains how I got it working to begin with. Now to update the image so it actually looks pretty.

Creating a Twitter card

I could have used a regular full-sized landscape image, but tbh I'll take any excuse to use Canva. I did warn you about the vanity.

That's my new favorite picture of Nova and me together <3

At this point I just needed to download it from Canva and add it to my static/ directory in my repo, and then update config.toml:

-  images = ["/nova-ears.png"]
+  images = ["/twitter-card-large.png"]

Now we can see it on Twitter!

The aspect ratio isn't quite right, unfortunately (Twitter's docs lied about it being 2:1). But it's a big improvement! And we still get the Nova ears logo. Hooray! 🎉

A User Journey: Setting Up the Node Beeline on Lambda

shelby spees (she/her) — Tue, 10 Nov 2020 22:50:53 +0000

Nic Wise at Tend Health recently wrote a series of blog posts exploring how they moved away from logs and metrics, toward adopting observability with Honeycomb. In that series, he shares lessons learned as they got their NodeJS app instrumented in an AWS environment making use of CloudFront, API Gateway, Lambda, and a few other services.

Tend is a New Zealand-based healthcare platform launching in 2020. With the opportunity of a near-greenfield project to work with, Nic shared his experiences learning Honeycomb that you might find useful.

Getting started

Nic used the Node Beeline integration and set up some middleware to get data into Honeycomb from dev, but encountered some snags with his lambda code waiting on API calls to Honeycomb’s ingest service. In order to solve that, he needed to better understand Honeycomb’s data model. He very clearly phrases his take on using events:

You can almost think of an event as a log line - something happened and I want to know about it - but it's also a data structure, so you can add any other context you want to it.

Once Nic updated his mental model of working with structured events, “around 90% of the metrics [he’d previously] created had no purpose anymore.” He changed his existing log lines and metrics to dump that data into the Beeline’s spans as context, deleting a bunch of code in the process.

Working with traces and AWS Lambda

Nic also encountered some missing spans because of how his codebase was using async/await:

It can be tricky to debug missing parts of a trace, but I can promise when you solve it you'll have learned that part of the code like the back of your hand.

That’s the thing about instrumentation and observability tooling: the point isn’t for the tools to think for you, it’s to help you better reason about your code and your systems. Auto-instrumentation is great for getting started but the most important parts of your code are unique to your app because they’re unique to your business. That means you’ll benefit the most from some adding custom instrumentation on top of auto-instrumentation, and it’s why Honeycomb makes the most sense as a developer tool.

Nic got advice from Honeycomb’s Pollinators community Slack on how to fix the issue with Lambda blocking on API calls. This involved dropping down into Libhoney, the event handler library used by the Beeline SDK, and redirecting his events to emit to stdout. From there, Cloudwatch Logs and Honeycomb’s agent Lambda integration send his data along to Honeycomb asynchronously.

Once he got that working, Nic encountered a scenario where he was seeing separate traces for an asynchronous call to the Twilio API, which he wanted to combine into a single trace alongside the caller. By updating his code to pass along the trace ID and parent span ID, he was able to combine them:

Nic’s scenario is common among teams using Lambda, but his post clearly explores that with a straightforward example.

Read Nic’s entire series plus his follow-up post to get the full story along with code samples:

Learn more about how metrics work in Honeycomb and how you can eliminate the majority of them by using structured events: download Getting Started with Honeycomb Metrics.

Join the swarm. Get started with Honeycomb for free.

HoneyByte: Get a Taste for Sampling

shelby spees (she/her) — Wed, 04 Nov 2020 22:42:38 +0000

Honeycomb's event-based pricing model is pretty simple: we only care about how many events you send. For teams running workloads at scale, the question becomes: are all of my events worth keeping? How can you reduce overall event volume while maintaining fidelity? This HoneyByte is all about sampling strategies you can use to lower costs without sacrificing the value of your data.

We'll look at several approaches you can use, help you avoid common pitfalls, and walk through real sampling code running in production (thanks to our friends at dev.to).

What is sampling?

Sampling is a way to reduce the amount of data you send to Honeycomb without a significant reduction in the quality of your data. It’s like getting samples of food: you can taste all the important bits without getting full.

Sampling requires making decisions ahead of time about what we’re going to include and what we’re going to leave out. If you’re new to the concept of sampling, you may want to start with Irving Popovetsky’s post on various downsampling strategies. For a deeper dive into why sampling is a good approach, you should start with this fantastic talk from Liz Fong-Jones:

You don't need to have read or watched those resources to understand the sampling concepts in this post, although I do presume enough familiarity with tracing that we can focus on implementing trace-aware custom sampling.

How does sampling work with Honeycomb?

Honeycomb does not downsample your data: we keep every event you send. However, Honeycomb can also receive your downsampled data and use each event’s sample rate to fill in the gaps and make your graphs look approximately as they would with the full data set. We keep your data, however you want to send it to us—it’s up to you.

Before we cover why samplers are implemented in particular ways, it’s important to be clear on how events are sent from your apps to Honeycomb. Previously, we’ve broken down how Honeycomb events are classified somewhat differently than how you might think about what a meaningful event means for your service. For example, with HTTP requests (which are usually represented as traces in the Honeycomb UI), each request creates one trace. Each trace (typically) is comprised of many spans. Each trace span typically represents an underlying unit of work (e.g. an additional system call that was made to fulfill the request, etc). To Honeycomb, each individual span you send is counted as one event. 1 span == 1 event.

An important thing to know is that your data isn’t batched at the trace level. Rather, spans are sent as individual events, and they’re only compiled into a trace by Honeycomb’s backend services once all of your data arrives.

Using a sample rate

The simplest way to downsample your events is by using a blanket sample_rate. The event volume you generate will be 1/sample_rate. In other words, a sample rate of 1 sends 1/1, or 100%, of your events to Honeycomb. A sample rate of 5 sends 1/5, or 20%, of your events. (Note: the sample rate must be an integer.)

For clarity, the default value for an event’s sample_rate is 1. A sample rate of 1 means that none of your data is downsampled in any of our Beelines or in Libhoney, by default.

Here’s an example Libhoney configuration borrowed from the Ruby Beeline docs, using a blanket sample_rate:

Honeycomb.configure do |config|
  config.client = Libhoney::Client.new(
    writekey: 'YOUR_API_KEY',
    dataset: 'ruby',
    sample_rate: 5
  )
end

How this plays out:

Libhoney sends 1 in every sample_rate events and drops the rest
In this case, it sends 1 in every 5 events and drops the other 4
Honeycomb receives the event, including its sample_rate field
Honeycomb’s backend uses the sample_rate to calculate what your overall dataset would look like
In this case, Honeycomb presumes that the 1 in 5 events sent is representative of the other 4 that were not

That last bullet point is important to note. What if the 1 event that was sent isn’t representative of the other 4? Dropping 80% of events across the board can be risky. It’s likely that errors or other interesting events aren’t going to be evenly distributed across your production traffic. For most scenarios, we recommend using custom sampling logic instead.

Overriding the sample hook

If you’re using the Ruby Beeline, you can implement custom sampling logic by overriding config.sample_hook in your Honeycomb Beeline configuration—where you include your API key.

Before worrying about various sample rates, let’s just look at how overriding the sample hook works with a toy example:

Honeycomb.configure do |config|
  # ... (other config settings, like write_key and dataset name)
  config.sample_hook do |fields|
    if fields["drop_me"]
      [false, 1]
    else
      [true, 1]
    end
  end
end

The config.sample_hook needs to return a list with two pieces of data for every event:

should I include this? (boolean)
`sample_rate` (integer)

So in the example, if I’ve added a field called drop_me to any span in my code, we’re returning [false, 1] for those events. In this case, false is answering the question, “should I include this event?” This type of approach can be useful for particularly noisy events with low-value data.

In my else clause, we’re returning [true, 1]. Here, true is saying, “yes, send this” and the sample rate of 1 tells Honeycomb that this event represents only itself, there’s no need to re-calculate missing events. (This toy example presumes we care not at all about the events we’re dropping! That’s rarely the case.)

The example implements our custom sampling logic directly in the config.sample_hook. But typically, we would want to write more sophisticated logic to decide which types of events we’ll be sampling. Let’s look at how to do that.

DEV sampler walk-through

In previous HoneyBytes, we looked at how the DEV team set up Honeycomb and how they gained observability by adding more context fields. Now, let’s look at how the DEV team implemented custom sampling in their code (recently renamed to forem/forem! Read their story.).

Starting from config/initializers/honeycomb.rb, where they have their Honeycomb configuration set up, they override config.sample_hook like we did in the above example:

Honeycomb.configure do |config|
  # ... (config stuff)
  # ...
  # Sample away highly redundant events
  config.sample_hook do |fields|
    Honeycomb::NoiseCancellingSampler.sample(fields)
  end
end

This time, though, they’re calling out to a custom sample method in their own Honeycomb::NoiseCancellingSampler class. First I’ll share the class implementation in full, and then I’ll walk through what each chunk of code is doing.

module Honeycomb
  class NoiseCancellingSampler
    extend Honeycomb::DeterministicSampler

    NOISY_REDIS_COMMANDS = [
      "GET rails-settings-cached/v1",
      "TIME",
    ].freeze

    NOISY_SQL_COMMANDS = [
      "BEGIN",
      "COMMIT",
    ].freeze

    NOISY_REDIS_PREFIXES = [
      "INCRBY",
      "TTL",
      "GET rack:",
      "SET rack:",
      "GET views/shell",
    ].freeze

    def self.sample(fields)
      rate = 1 # include everything by default
      # should_sample is a no-op if the rate is 1

      if fields["redis.command"].in? NOISY_REDIS_COMMANDS
        rate = 100
      elsif fields["sql.active_record.sql"].in? NOISY_SQL_COMMANDS
        rate = 100
      elsif fields["redis.command"]&amp;.start_with?("BRPOP")
        # BRPOP is disproportionately noisy and not really interesting
        rate = 1000
      elsif fields["redis.command"]&amp;.start_with?(*NOISY_REDIS_PREFIXES)
        rate = 100
      end
      [should_sample(rate, fields["trace.trace_id"]), rate]
    end
  end
end

To start, we need to extend the Honeycomb::DeterministicSampler module in the Ruby Beeline. The important thing to know about this module is that we're using it in order to call its should_sample method down in line 37, which decides whether to keep or drop each event based on the sample rate provided. I’ll explain that a bit more later, when we get to that line.

The DEV team had found that Redis and SQL queries were generating a lot of noisy events that weren’t very useful. So they made a few lists of noisy commands to filter by. Here’s the list for Redis:

NOISY_REDIS_COMMANDS = [
  "GET rails-settings-cached/v1",
  "TIME",
].freeze

We can see in Honeycomb's trace view that the TIME command shows up a lot:

TIME is not very interesting, so it makes sense that they would want to downsample it. It’s the same idea for the other commands they’re downsampling.

Now let’s walk through the sample method, copied here:

def self.sample(fields)
  rate = 1 # include everything by default
  # should_sample is a no-op if the rate is 1

  if fields["redis.command"].in? NOISY_REDIS_COMMANDS
    rate = 100
  elsif fields["sql.active_record.sql"].in? NOISY_SQL_COMMANDS
    rate = 100
  elsif fields["redis.command"]&amp;.start_with?("BRPOP")
    # BRPOP is disproportionately noisy and not really interesting
    rate = 1000
  elsif fields["redis.command"]&amp;.start_with?(*NOISY_REDIS_PREFIXES)
    rate = 100
  end
  [should_sample(rate, fields["trace.trace_id"]), rate]
end

We set the default rate to 1, which means including 100% of events.

From there we set the rate to different values depending on the events fields:

send 1% of noisy Redis commands
send 1% of noisy SQL commands
send 0.1% of `BRPOP`, which is especially noisy in Redis
send 1% of Redis commands with noisy prefixes

As I explained in earlier sections, the sample hook calling the custom NoiseCancellingSampler.sample method here is expecting a tuple, where the first element answers “is this sampled?” and the second element is the sample_rate. In this case the tuple in our return statement is answering the first question by calling should_sample(rate, fields["trace.trace_id"]). This is the should_sample I mentioned earlier—it’s what we’re leaning on to make consistent sampling decisions, and it’s why the NoiseCancelingSampler class extends the Beeline’s DeterministicSampler module at the beginning.

The should_sample method expects a rate and a value, and returns a boolean. Note: a rate of 1 always returns true (i.e., “keep this event”). Beyond that, should_sample will give a consistent return value with consistent inputs. This is why we pass in the trace.trace_id field—so that we get the same result for all spans within a trace. That’s the key to trace-aware sampling!

Returning the tuple completes the implementation of the DEV team’s custom sample_hook. The Beeline then takes the resulting boolean value and the rate, sending events with true over to Honeycomb along with the corresponding sample rate, and dropping all the events with false.

To step back a bit: you can implement whatever logic you want for your custom sample hook, as long as it returns that tuple: [boolean, rate]. And your boolean value here is generated from calling should_sample, which is made trace-aware by passing in the trace.trace_id. The rest is up to you!

Another nice thing about using the Beeline’s DeterministicSampler is that it’s just code, which means that you can write tests for it! Check out the RSpec tests for the DEV custom sampler.

A template custom sample hook

To recap, I’ve written out a complete example that you can use as a template for your own custom sampling logic. First I overwrite config.sample_hook by calling out to MyCustomSampler.sample:

Honeycomb.configure do |config|
  # ... (config stuff)
  # ...
  config.sample_hook do |fields|
    MyCustomSampler.sample(fields)
  end
end

And here’s my custom module that extends the Beeline’s DeterministicSampler, implements my sampling logic, and then calls should_sample using the trace ID to keep things consistent, returning our expected tuple:

module MyCustomSampler
  extend Honeycomb::DeterministicSampler

  # keep 1% of anything with `downsample` set to true
  def sample(fields)
    rate = 1
    if fields["downsample"]
      rate = 100
    end
    [should_sample(rate, fields["trace.trace_id"]), rate]
  end
end

One thing to note is that I’m setting the downsample field elsewhere, in my actual application code. You don’t have to use a special field to decide what to sample on, you can sample on an event’s name field or based on values of specific fields like the DEV team did in their code. Think about what’s important vs. what’s noise—you know your code best!

Common sampling pitfalls

The logic we’ve described so far is great for individual events, and it makes a lot of sense for the DEV team to downsample noisy Redis or SQL commands that don’t provide a lot of information and don’t have any child spans.

Honeycomb knows to re-calculate query results for individual dropped events based on the sample_rate of their counterpart events that do get sent. But! Honeycomb doesn't know where those missing events would go in the trace waterfall. The dropped spans aren't there to establish the parent/child relationship needed to render the spans in the right locations. The DEV team is aware of this trade-off—they are purposely dropping leaf spans (those with no children) in order to reduce event volume. If your sampling code drops spans with children, however, your trace waterfall will show that you're missing spans in the middle. That's probably not what you want.

There are a few more considerations to keep in mind when working with traces. Let’s look at what happens when our code makes sampling decisions without being trace-aware. Here we’re trying a head-based sampling approach to drop a trace at the request level:

def index
  if not_interesting
    Honeycomb.add_field('downsample', true)
  end
  # ... (render page, etc.)
end

We’re setting the downsample field based on some condition in the scope of the index method. Unfortunately it’s gone wrong because we’re trying to downsample at the request level, but what happens is that only the root span is dropped while all the child spans still get sent. Here’s what that would look like in the trace view:

The solution for avoiding this is to set a trace-level field as early as possible, before any child spans are started. Our sampling hook will then check the trace-level field on each event to decide whether it gets sent:

def index
  if not_interesting
    Honeycomb.add_field_to_trace('downsample', true) # added to the whole trace
  end
  # ... (render page, etc.)
end

Another common pitfall occurs with tail-based sampling. With tail-based sampling, the decision to drop an entire trace is based on some value that is set in the middle of a request. If that happens, the result can lead to orphaned child spans, which get sent to Honeycomb before the entire request completes and the trace-level data is sent.

Note in this screenshot, both the root span and child 2 were dropped. child 1 and grandchild were already sent to Honeycomb by the time the trace-level field for dropping events gets set by child 2.

The logic to complete the sophisticated batching required for tail-based sampling is not built into our Beelines or Libhoney. The best approach to do tail-based sampling without breaking your traces is to run your events through a proxy that can buffer full traces before downsampling them and sending them to Honeycomb. Stay tuned for future news on ways to set up these proxies.

How will you sample events?

We hope this tutorial sheds light on how you can build sampling logic into your code. Check out our docs for more guidance on sampling. You can also check out the source code used to generate these examples, along with more detailed explanations of how things went wrong. Finally, learn more by downloading our white paper, The New Rules of Sampling (direct PDF download).

Have questions? Missing spans? Reach out to our support team, or join the Pollinators Slack community.

Join the swarm. Get started with Honeycomb for free.

Interview with Honeycomb Engineer Chris Toshok: Dogfooding OpenTelemetry

shelby spees (she/her) — Mon, 26 Oct 2020 17:27:35 +0000

At Honeycomb, we talk a lot about eating our own dogfood. Since we use Honeycomb to observe Honeycomb, we have many opportunities to try out UX changes ourselves before rolling them out to all of our users.

UX doesn’t stop at the UI though! Developer experience matters too, especially when getting started with observability. We often get questions about the difference between using our Beeline SDKs compared with other integrations, especially OpenTelemetry (abbreviated "OTel"). That’s why the team decided to do some integration dogfooding by instrumenting our own code with OpenTelemetry alongside existing Beeline instrumentation.

Poodle is the frontend Go service that renders visualizations in the Honeycomb UI after getting query results and traces from the backend. Engineer Chris Toshok has been working on adding OpenTelemetry to the Poodle code and comparing it with our existing Beeline integration. I talked to Chris about his experience setting up OpenTelemetry from the perspective of a practitioner and service owner.

Interview with Chris Toshok

What were your thoughts going into the effort?
The main concern was schema compatibility. We have existing triggers, boards, SLOs, other things that relied on the schema that the Beeline generated. If we only sent to the existing Poodle dataset, we’d have to make sure that the OTel data would end up with the same field names and values as what's already there, so it wouldn’t break the things that depend on the existing schema.

Alternatively, we could double-send: the Beeline data goes to one dataset and OTel data goes to another. We ended up going this way so that we can look at what the differences are between the two schemas without breaking existing dependencies in our Dogfood team.

Was there anything that surprised you in the process?
While there are lots of little differences between the APIs, the core concepts were the same. The only real sort of surprise was that the Beelines actually let you do stuff that’s kind of scary, because of how the data is stored.

With OTel, as soon as set a value for a field, you can’t modify that value, only replace it. In the Beeline, it’s just a reference to a Go struct. You can just change the values at any point before the data gets sent to Honeycomb. With some fields, like the team object within a span, changing that could be dangerous. But sometimes you want to change a certain value in the middle, like if you need to sanitize values before sending them to Honeycomb.

The configuration for OTel is a bit different from the Beeline, did you also work on getting set up with that?
We basically just set up the OpenTelemetry-Honeycomb Exporter for Go, written by engineer Alyson van Hardenberg.

In the code, I used dependency injection to create an OTel-compatible initializer that works similarly to the Beeline. Beeline data and OTel data are stored a bit differently before they’re sent out, so we need to handle both cases. I added a shim that looks very similar to the OTel API, to wrap around both OTel and the Beeline and allow us to send to Honeycomb using either or both.

If you could give advice to a team starting today, would you recommend they use the Beeline or OTel for their Go app?
Beeline has a much more specific API that maps more closely to Honeycomb’s features. OTel is doing things at a more general layer, so it might not have all the interesting application bits that we have after several years of doing Honeycomb-specific work in the Go Beeline, which is an effect of the Beeline being something we built out alongside Honeycomb itself. We developed it to answer the questions we had about the service, and to make use of Honeycomb’s features.

What sort of differences have you found?
There were a couple small, but notable differences. OTel sends JSON blobs as strings, which don’t work with the JSON unfurling feature in Honeycomb.

Also, Beelines allow us to set trace-level attributes. Any span that we send can add a trace-level field. OTel doesn’t support trace-level fields added from child spans.

Have you noticed any performance changes in Poodle after adding OTel?
No change, but I’m wishing we had time over time queries. This is something I was interested in checking on and seeing in time series form.

Observing the impact

Toshok shared with me the queries he ran to check for any impact to Poodle's performance. Here's the baseline behavior from the week before Toshok's change went out:

Here's the same query, run on the following week. Toshok's changes got rolled out on 7/21:

It makes sense that there would be minimal impact in this case because of how Toshok implemented the change, but it's still neat to be able to query for it.

Try it yourself!

Interested in comparing OTel and Beeline? Here’s how Toshok did it. For the most part they were 1-1 changes using the new OTel-aligned shim API:

- span.AddField("error", err.Error())
+ span.SetAttribute("error", err.Error())

Occasionally Toshok needed to do some type manipulation for compatibility with OTel:

- beeline.AddField(ctx, "error", err.Error())
+ span.SetAttributes(instr.AppString("error", err.Error()))

But the overall shape of the instrumentation remained the same:

- ctx, span := beeline.StartSpan(ctx, "runQueries")
- defer span.Send()
+ ctx, span := instr.StartSpan(ctx, "runQueries")
+ defer span.End()

If you're starting out from scratch, follow our OpenTelemetry Go tutorial to get up and running with Honeycomb in no time!

Join the swarm. Get started with Honeycomb for free.

Get on the Right Track with Our Rails Integration!

shelby spees (she/her) — Wed, 21 Oct 2020 17:42:20 +0000

Thanks to awesome contributions from the community and the hard work of our integrations team, the Honeycomb Rails integration comes with lots of great features out of the box. We’ve written about Rails-specific features of the Ruby Beeline before on the Honeycomb blog:

This post is an end-to-end tutorial to show you exactly the steps involved, from creating a new Honeycomb team to getting your data in and observing your app in production.

Whether you’re starting a new Rails project and want to build observability in from the start, or you’re investing in better observability for your existing Rails service, this tutorial will help you get the most out of Honeycomb right away.

Create a free Honeycomb account

Go to ui.honeycomb.io/signup to create your account. We support SSO with Google, which means no new passwords to create or remember.

Provide your organization’s name when prompted. This is a unique ID, so you can either use your company's name, or if it's a large company you might go with company-team, e.g. huuli-research.

If you're prompted to name your first dataset, call it rails-app or give it the same name as your app. (If you change your mind about the name later, you can just create a new dataset.) Select the Ruby integration and click Create.

Once you’ve created your team, you’ll automatically receive an email inviting you to the Honeycomb Pollinators Slack community. This blog post from Customer Advocate Jenni Boyer details why you should join, definitely check it out when you get the chance!

Get your API key

The next page will give you your Honeycomb API key (also called the write key). If you chose the Ruby integration, this page will also list out instructions for setting up the Ruby Beeline, including Rails-specific instructions. Don’t worry about them, though—I‘ll break down each step for you here.

Copy your API key now, we’ll be using it in the next steps. (If you lose track of your API key, you can always copy it again from your Team Settings.)

Configure the Ruby Beeline SDK

Now you can open up your terminal and set up the Rails integration for your app. The Ruby Beeline SDK comes in the form of a gem, so we’ll add it to our Gemfile using Bundler:

$ bundle add honeycomb-beeline

Now we’re ready to generate the Honeycomb configuration. The Beeline has a Rails generator built-in, so we can use that with the API key you copied earlier (denoted abc123 here) along with your dataset’s name, rails-app:

$ rails generate honeycomb abc123 --dataset rails-app

This will create a new file: config/initializers/honeycomb.rb. The generator hard-codes your API key to start, but we’ll update this file later to store your key more safely.

Generate Your First Events

You’re ready to start sending data to Honeycomb! Start the Rails server and then navigate to localhost:3000 in your browser, or run curl localhost:3000 in your terminal if you prefer. This will hit the root path and send your first event to Honeycomb. Let’s return to ui.honeycomb.io to watch it arrive!

The home page shows us a single data point on the Total Requests graph and a single trace under Recent Traces. Success!

With just this one request sent from dev, there’s not enough data to make the home page very interesting. But we can still poke around a bit. Click the button on your trace under Recent Traces.

This takes us into the trace view for the request we just made:

The tracing we see here comes built in with the Ruby Beeline! Not only do we automatically get spans wrapped around top-level HTTP requests, ActiveRecord calls, controller actions, and view rendering, we also get relevant context fields attached to all of those spans. Select the root span http_request and scroll through the Fields list in the right-hand sidebar to see what fields are attached to that span. Then choose another span, like process_action.action_controller, to see the fields that come included with that one.

You may already be thinking of additional context fields to instrument various parts of your app to send to Honeycomb, but this is a good stopping point. What are the next steps to get this MVP version into production?

Get Ready to Ship

Your app is now instrumented for Honeycomb! Let’s update a couple things so you can open a PR, push your new instrumentation to prod, and start unlocking more insights.

Earlier, you called the rails generate command with your API key directly, creating the config/initializers/honeycomb.rb file. Now’s the time to remove the hard-coded API key there and store it safely according to how your team manages secrets.

For example, if you’re using the Rails config/secrets.yml file to manage your secrets, you can add your API key there.

development:
  honeycomb_write_key: 'abc123'
production:
  honeycomb_write_key: 'abc123'

Then, update the Honeycomb initializer to safely use the API key straight from the secrets file:

- config.write_key = "abc123"
+ config.write_key = Rails.application.secrets.honeycomb_write_key

And you’re set! Open that PR 🚢 (and check out the Pollinators blog post I shared earlier while you're waiting for PR reviews 😉).

Observe Production Traffic

Why the rush? Honeycomb’s home page comes alive in production. We can see the Total requests, Latency, and Error rate using the Rails auto-instrumentation built into the Ruby Beeline.

Clicking over to the HTTP Status Code tab, we can also see the breakdown of response codes as well as the distribution of latencies for each one.

There’s a ton to explore here, just start clicking around. See how much you can learn from the auto-instrumentation by itself.

Add Custom Instrumentation

Now that you’ve explored Honeycomb with your app’s production traffic, it’s time to dig deeper. You have a handle on events and tracing for regular Rails HTTP requests, so let’s try adding custom spans and fields. What’s unique about your app that you’d like to learn more about?

You likely have some home-rolled lib/ code that’s not covered by the Beeline’s auto-instrumentation. For example, imagine that your app uses the twitter gem to get a user’s tweets in batches of 200 at a time:

def get_all_tweets(user)
  all = []
  options = { count: 200, include_rts: true }
  loop do
    tweets = client.user_timeline(user, options)
    return all if tweets.empty?
    all += tweets
    options[:max_id] = tweets.last.id - 1
  end
end

Here’s an updated version with some instrumentation:

def get_all_tweets(user)
  Honeycomb.start_span(name: 'get_all_tweets') do
    Honeycomb.add_field_to_trace('user', user)
    all = []
    options = { count: 200, include_rts: true }
    loop do
      Honeycomb.start_span(name: 'get_batch') do
        Honeycomb.add_field('options', options)
        tweets = client.user_timeline(user, options)
        return all if tweets.empty?
        all += tweets
        options[:max_id] = tweets.last.id - 1
      end
    end
  end
end

We start by wrapping the entire get_all_tweets method body in a span, and then we add the user field to the entire trace, since we’ll probably care about it across the entire request. We add a second span block around the body of the loop called get_batch, which creates a child span for each request where we get a 200-tweet batch. Inside of get_batch, we’re adding the request options as a field as well, so we can see the relevant arguments when hitting Twitter’s API.

(At this point I’d probably break out the loop body into a separate get_batch method, which is usually about the scope you want for a span.)

Our nested spans will be rendered as a trace in the Honeycomb UI:

Each get_batch is making an API call to Twitter, so to get all of the tweets from this user we needed to make eight API calls.

Continue Your Journey!

The Ruby Beeline does a lot of the heavy lifting to get you sending rich data into Honeycomb quickly and without a lot of work. On top of that, it’s not a lot of code to start adding your own custom instrumentation beyond what the Beeline does automatically.

Now that you have data in, invite your team and see what insights you can learn together! Check out this blog post on helpful Honeycomb features for onboarding your teammates.

What's your favorite incident story?

shelby spees (she/her) — Mon, 19 Oct 2020 15:51:26 +0000

How was the issue discovered? How were you involved? What was the impact? What did you learn?

Have you ever been on-call? What was it like?

shelby spees (she/her) — Thu, 15 Oct 2020 17:39:42 +0000

Would you do it again? What would you change?

Observability 101: Terminology and Concepts

shelby spees (she/her) — Wed, 14 Oct 2020 19:01:11 +0000

When I first started following Charity on Twitter back in early 2019, I was quickly overwhelmed by the new words and concepts she was discussing. I liked the results she described: faster debugging, less alert fatigue, happier users. Those are all things I wanted for my team! But I was hung up on these big polysyllabic words, which stopped me from taking those first steps toward improving our own observability.

This post is my attempt to help orient folks who want to learn more about observability but maybe feel overwhelmed or intimidated by the vocabulary list, like I did. My goal is to get everyone on the same page about what these words mean so that we can focus on leveraging the tools and ideas to build better software and deliver more value! 💪

Welcome to Observability 101! 🤓

Observability Fundamentals

In software, observability (abbreviated as “o11y”) is the ability to ask new questions of the health of your running services without deploying new instrumentation. The term comes from control theory, which defines observability as the ability to understand the internal state of a system from its external outputs (source).

Telemetry consists of those “outputs”—it’s the data generated by your system that documents its state. Telemetry gets generated because of instrumentation: code or tooling that captures data about the state of your running system and stores it in various formats. Some examples of software telemetry include: metrics, logs, traces, and structured events.

Modern software teams have gotten good at accounting for failures that can be caught by tests and continuous integration tooling. We use retries and autoscaling and failovers to make our systems more resilient in the wild world of production. Since we catch and respond to known variables so well, what we're left with are the unknown-unknowns. The types of issues we often see in modern production software systems are emergent failure modes, which happen when a bunch of unlikely events line up to degrade or take down your system. They're really interesting but difficult to debug, which is why we need observability.

In order to have good observability into your software services, you need both good instrumentation generating high-context telemetry data, and you need sophisticated ways of interacting with that data that enable asking novel questions—questions you couldn’t have thought of when you wrote the code. Put more simply, software observability requires good data and good tooling. Let’s start by discussing the data.

Software Telemetry

To reiterate, telemetry is data that your system generates that tells you about the system's health. The term comes from the Greek tele-, meaning "remote", and -metry, meaning "measure." You probably already generate telemetry from your programs, even if you’re not paying for monitoring or logging services. In fact, even the output from a print or console.log() statement is a form of telemetry!

Let’s discuss the most common forms of telemetry generated from production software services.

Metrics

One commonly used form of telemetry data in software is metrics. A metric consists of a single numeric value tracked over time. Traditional monitoring uses system-level metrics to track things like CPU, memory, and disk performance. This data is important for choosing among virtual machine instance types, with options for processor speed, RAM, and hard disk storage. But it doesn’t tell you about user experience, or how to improve the performance of your code.

Modern monitoring services also provide application performance monitoring (APM) features, which track application-level metrics like average page load times, requests per second, and error rates. Each metric only tracks one variable, which makes them cheap to send and store. Values are pre-aggregated at write-time, however, so you need to deploy a code change if you want to track metrics for a new intersection of data, e.g. error rates for a specific user.

Logs

Logs are text strings written to the terminal or to a file (often referred to as a “flat” log file). Logs can be any arbitrary string, but programming languages and frameworks have libraries to generate logs from your running code with relevant data at different levels of specificity (e.g. INFO vs. DEBUG mode). There’s no standardization among programming communities about what should get included at each log level.

Log aggregation services allow you to send the content of your logs and store them for later retrieval. Querying flat logs (as opposed to structured logs) is slow because of the computational complexity of indexing and parsing strings, which makes these tools impractical for debugging and investigating your production code in near-realtime.

Traces

A trace is a visualization of the events in your system showing the calling relationship between parent and child events as well as timing data for each event. The individual events that form a trace are called spans. Each span stores the start time, duration, and parent_id. Spans without a parent_id are rendered as root spans.

A distributed trace connects calling relationships among events in distributed services. For example, Service A calls Service B, which makes a database query and then hits a third-party API.

Structured Events

A structured event is a data format that allows you store key-value pairs for an arbitrary number of fields or dimensions, often in JSON format. At minimum, structured events usually have a timestamp and a name field. Instrumentation libraries can automatically add other relevant data like the request endpoint, the user-agent, or the database query.

When we talk about wide events, we mean structured events with lots of fields. You can add whatever fields you want, and because they’re key-value pairs, it’s easy to query for them later on.

Events can be built into traces by pointing each event to its parent event with the parent_id field.

Telemetry concepts

These are some more terms commonly used when discussing observability.

Context refers to the collection of additional dimensions, fields, or tags on a piece of data that tells you more about the internal state of your system. Context can include metadata, system-level metrics, and domain-specific information about the running program.

The dimensionality of data refers to how many fields or attributes a piece of data has attached to it. Examples of low-dimensionality data include metrics and flat logs. High-dimensionality data includes anything that can support many fields or attributes, including JSON blobs, structs, and objects.

The term cardinality refers to how many possible values a dimension can have. Some examples of low-cardinality data include booleans (true/false) and days of the week. High-cardinality examples are first name, social security number, UUID, or build ID.

Cardinality becomes becomes especially relevant when you start considering multiple dimensions at the same time. For a REST API, for example, endpoints can number from a dozens or hundreds (not including unique IDs). If you want to look at the behavior of each endpoint across the over 43 million user agents in the world, you get what’s called a cardinality explosion. Imagine trying to track the intersection of endpoint × user agent with application-level metrics—that’s billions of lines of code. Thus when you have questions that require high-cardinality data to answer, you need telemetry and tools that support it.

Instrumentation with Honeycomb

Telemetry gets generated from instrumentation, which is code or tooling that gathers data from your system in real-time. Whatever observability tool you’re using, there’s some set up and configuration to do in order to generate telemetry data and then send it off to your vendor or service.

For brevity and clarity, I’m limiting this section to Honeycomb-specific instrumentation.

Instrumenting code for structured data

At Honeycomb we feel that you get the best observability into your systems when you instrument your code. There are a number of ways you can do that.

You can roll your own structured data by generating structured logs using your language or framework’s existing logging libraries. To get your structured logs into Honeycomb, this approach requires first instrumenting your code and then deploying an agent that’s configured to send the data to Honeycomb.

There are also open-source options with built-in support for structured data. OpenTelemetry (abbreviated “OTel”) is an open-source, vendor-neutral integration that can be used to instrument your service to generate structured telemetry data. OpenTelemetry is available as a package in a number languages--Go, Java, .NET, JavaScript, Python, Ruby, and others--with auto-instrumentation available for many popular frameworks. Also, since OpenTelemetry is vendor-neutral, you can use it to send the same telemetry data to multiple services!

Honeycomb was founded before OpenTelemetry existed, however, so we started out by writing our own instrumentation libraries—Beelines—to help you get data into Honeycomb. Our Beelines are available as a package in five languages: Go, Java, JavaScript, Python, and Ruby. The Beelines have auto-instrumentation for popular libraries and frameworks, including each language’s HTTP library as well as Spring, Rails, Flask, Express.js, and others.

Under the hood, the Beeline integrations use Libhoney, our low-level event handler SDK in each of the languages. Libhoney does not support tracing out of the box, and it’s rare that you should need to interface with it directly. If you’re just getting started with Honeycomb, we recommend using a Beeline or OpenTelemetry. The Get Your Data In docs provide more guidance on choosing an instrumentation library.

What observability tooling needs

Observability goes beyond application-level monitoring. It goes beyond tracing. It goes beyond “three pillars.”

Observability tooling needs to support structured events, unaggregated data blobs containing whatever key-values pairs you decide to send. It needs to empower you to flow seamlessly between broad aggregates (time series graphs) and deep context (trace visualizations) and then broad aggregates again, using your structured events’ context fields to ensure that the answers you get are relevant and informative. Observability means aggregating at query-time, not at write-time, because you never know what questions you might want to ask later on.

With observability, you can deeply interrogate your code behavior in production because you have more than just isolated time series graphs and some trace visualizations. You can send whatever fields might be relevant inside your structured events, and then ask new questions that your vendor (and you!) could never anticipate.

Reach out!

Have questions? Notice something I missed? Reach out to me on Twitter, or email me directly at shelby@honeycomb.io. I also hold office hours—schedule 30 minutes with me so I can help you reach the next level on your observability journey!

Experience what Honeycomb can do for your business. Watch this short demo!

HoneyByte: Incremental Instrumentation Beyond the Beeline

shelby spees (she/her) — Wed, 15 Jul 2020 03:56:40 +0000

"It turns out," said Liz, "it was not a giant pile of work to start adding those rich instrumentation spans as you need them."

.ltag__user__id__341241 .follow-action-button { background-color: #0D4D4B !important; color: #fdf9f3 !important; border-color: #0D4D4B !important; }

Liz Fong-Jones

Principal Dev Advocate @honeycombio, SRE, Observability expert

Liz Fong-Jones was telling dev.to’s Molly Struve about an error she encountered while trying to update her dev.to profile. When she entered honeycomb.io into the Employer URL field, the app responded with an angry red box:

dev.to’s URL validator is quite picky--it rejects anything not prepended with http or https. Most users won’t think to include that, however. Liz certainly didn’t.

Faced with this error message, Liz thought, "Gee, thanks. That’s really not helpful." Then she removed her frustrated user hat and exchanged it for her engineer hat. "I was really really curious. How often is this happening?"

In a previous HoneyByte, we showed how Molly Struve integrated the Ruby Beeline with dev.to's Rails codebase in one step.

Leaning on the Beeline integration, which already wraps the app’s requests in spans, Liz opened a pull request on GitHub with two changes to the update method of the users_controller.

First, she added an error field so that the user-facing error message gets added to the current span’s context:

Honeycomb.add_field("error", @user.errors.messages)

Then she made sure the update method returned a 400 BAD REQUEST for any invalid user input, which is the reason for ending up in that else clause:

render :edit, status: :bad_request

Once her PR got merged, Liz reproduced the error by making the same edit to her dev.to profile Employer URL, returning the same error message. Then she went into Honeycomb to validate that the new contexts fields got sent along with her request:

Great, the PR was a success!

Catch All the Errors!

It turns out, Liz's pull request didn't just add instrumentation for the Employer URL field. Now dev.to is sending new context for any update request where a user encounters an error.

Liz wanted to see what other errors might be interesting. Since she streams on Twitch, she queried for all instances of invalid Twitch URL input. What she found was surprising.

dev.to users were entering their Twitter URLs into the Twitch URL field! After seeing the data, the user flow jumps out: People quickly scrolling down to fill out profile fields could easily mix Twitch with Twitter.

"When I was solving the 'I can't update my employer URL,'" said Liz, "I did not anticipate being able to debug Twitch URLs and any other field that gets added in the future to the users page. But now that's been done with two lines of code."

There are numerous design solutions for this bit of friction, but it's likely that it never would have been discovered without Liz's change.

Compare with Traditional Metrics

Let’s imagine investigating this trend using time series metrics.

For those who have tried to instrument your return status codes like what Liz did, you probably saw a number of 400 errors in production and scratched your head, unable to connect the errors to the code that sent them. Invalid user input doesn’t throw an exception, so it won’t be caught by error monitoring tools. You’d never learn about users confusing the Twitch URL field with Twitter URL.

Let’s say your CEO found the Twitch/Twitter issue while trying to edit their own dev.to profile and specifically asked you to find out how frequently it occurs.

In order to track the Twitch URL issue, you need to count that error message in isolation. It’s relatively manageable to add statsd metrics for the dozen or so input fields on a dev.to profile, but you may decide to only count the twitch_url if that’s all you’re being asked about.

if @user.errors.include? :twitch_url
  $statsd.increment("error.twitch_url")
end

Without your CEO’s specific question, you have to decide in advance to send error messages in the first place. You might count every kind of error in your model:

@user.errors.keys.each do |key|
  $statsd.increment("error.#{key}")
end

In Rails we can take for granted that the error keys are limited by the columns in the User model, but in a different context it can be risky to generate metric names dynamically. If you have unpredictable error names and create statsd counters on the fly, you’ll end up with an unbounded explosion of metrics, each with only a few occurrences. In the mean time, you’re being charged for each one. That cost adds up quickly, which results in teams that are averse to sending data.

On top of that, adding counters for errors only gets you so far. Unless you’re also tracking the parameter that was considered invalid, there’s no context that’ll tell you that users are mixing up the Twitch URL input field with Twitter.

If, for some reason, you actually thought to send the invalid parameter as its own metric (unbounded high cardinality be damned), you still have to do the work of matching the bad input to the 400 errors! It’s painful to query for this in time-series graph visualizations. You’re squinting at disparate graphs, each with a single occurrence of the thing you care about. At the end of the day, you’re just seeing aggregate counts and trying to reverse-engineer the user flow.

The Twitch/Twitter URL issue will likely remain invisible, even with the granular metrics you’ve added. Understandably, most teams won’t bother.

Compare this with Liz's instrumentation. By adding a single field, you get rich data for a variety of contexts. No counters necessary. You gain observability into friction in your user experience that doesn't get surfaced in bug reports. This opens the door for granular A/B testing, feature flagging, and usability research.

Your Turn!

If you already use Honeycomb, you probably already have a commonly-tread path in your codebase that you've instrumented, especially if you’re using one of the Beelines. Look for somewhere in your business logic where you return an error that isn’t an exception. For example:

bad user input
permissions error
network error

What context are you currently sending to Honeycomb? Are there more fields you can add? Don’t be afraid to send wide events with many context fields. Honeycomb is designed to handle it!

Want to learn more about observability? Download the Honeycomb Guide to Achieving Observability today!

HoneyByte: Make a Beeline Toward Observability Just Like DEV’s Molly Struve

shelby spees (she/her) — Mon, 13 Jul 2020 19:59:01 +0000

“When things broke,” Molly explained, “you’re mad scrambling—jumping from website to website to website, trying to put the pieces together.” Molly was able to use Honeycomb to fix things up: “It makes my job easier as an SRE.”

.ltag__user__id__119473 .follow-action-button { background-color: #61122F !important; color: #FFFFFF !important; border-color: #61122F !important; }

Molly Struve (she/her)

International Speaker 🗣 Runner 🏃‍♀️ Always Ambitious. Never Satisfied. I ride 🦄's IRL

Getting started with Honeycomb doesn’t require a lot of work: at dev.to, they used the Ruby Beeline to get it going: “I didn’t do that much,” she said. “I feel like I cheated!” We invite you along on their journey, and we'll share precisely what lines of code dev.to used to get started.

Service Fatigue

“When I started at DEV, there was very little in the way of monitoring, in terms of observability,” said Molly. “We had six or seven little tools, each doing one little piece.”

While each tool had initially served a purpose for the team, the fact that there were so many became a pain point during incidents. It’s a familiar experience for SREs or anyone who owns code in production.

“You’d look at one service, you’d see a spike, but that was all the information you'd get. So then you’d have to go to the other services and try to line up when the spike happened.”

Even during normal site operations, that’s enough to fatigue most people. With the added stress of an outage it becomes painful, error-prone, or both. That’s why Molly took the initiative to consolidate DEV’s use of monitoring services as much as possible. Whether facing an outage or just wanting to see how the site was being used, she wanted all that information in one place.

Molly’s initial goal was to consolidate her tooling, but what she found with Honeycomb were benefits beyond easing the immediate pain of service fatigue.

“Honeycomb really gives us some great insight into how users are interacting with our platform,” she said. “It’s been super helpful not only for debugging issues, but allowing me to find places in the application where slowdowns are happening, we’re hitting the database too many times, etc.”

It’s no coincidence that this was Molly’s first project after joining the DEV team. She gave a talk at SREcon19 last year on the benefits of an easy-to-use, centralized monitoring system. “The ability to have all that information in one spot and then be able to really drill down is super convenient,” she said. “Having it all in one place has been super helpful.”

Additionally, streamlining their tooling has simplified onboarding engineers at DEV. “They no longer have to learn five, six, seven different tools,” she explained. “When they come on, I can say 'Here’s Honeycomb, this is how you use it,' and that’s it.”

“Sometimes people think when you’re an SRE that we just magically know, but we don’t,” said Molly. “Half of what we’re doing is literally just throwing in queries, hitting stuff. We look at a graph, ‘Ah there’s nothing there.’ Okay let’s look at a different one, ‘Ah there’s nothing there.’”

Molly makes videos for her teammates demonstrating how she uses Honeycomb to help them learn the tool, and seeing her exploration process has helped them understand that it’s more about mindset than expertise.

“I think that was a big aha moment for some people,” she said. “There’s no shortcuts to what we do, it’s basically getting all the data and finding the pieces that stick out.”

Honeycomb enables first-principles debugging. That's when you don’t have to be an expert in the system, you just need to know where to start digging. Demystifying debugging as a process can empower an entire team.

Getting Up and Running

Molly described her experience adding Honeycomb’s Beeline gem for Ruby. It was simpler than she anticipated.

“I didn’t do that much,” she said. “I feel like I cheated!”

“I literally just took your documentation, slapped it into the project and it was like, Boom! there’s all of your data,” she said. “It was pretty awesome.” DEV is open-source, so you can see the exact change she made in her pull request on GitHub:

add honeycomb-beeline gem to Gemfile
add Honeycomb API key to secrets
generate Honeycomb configuration using the Beeline’s Rails generator: bundle exec rails generate honeycomb YOUR_API_KEY --dataset rails

Once Molly’s PR got merged, the DEV team was sending data to Honeycomb and gaining new insights.

“It was super straightforward, and honestly that got us probably 80% of the way there,” said Molly. “Any tool where you can get up and running so quickly is huge.”

By using the Ruby Beeline first, Molly reduced the overhead of starting out compared to instrumenting the entire application from scratch. “If you can get value out of it before you even have to customize? That’s a pretty big deal. If you have to go into the weeds before you can get value? That’s a blocker for a lot of people.”

Once she used Honeycomb with production data, Molly thought of specific areas she wanted to instrument in more detail. Observability isn’t all-or-nothing. The incremental approach is a lot less scary.

Using the Tool

Honeycomb has several features that encourage users to dive in and start querying. The suggested queries on the home page help users get oriented.

“That’s one of the first things I start most of the videos with,” Molly said, referring to the instructional videos she makes for her teammates. Looking at the home page, Molly said, “something sticks out and then I kind of dive into it.”

Another feature of Honeycomb that Molly appreciates is the query builder, in particular, the autocomplete. “That helps immensely in trying to figure out what fields you want to query for.”

Rather than having users type into a blank query editor, the query builder reduces friction by breaking up clauses into separate fields. There’s no bespoke query language to learn, it’s SQL but more visual and with suggestions.

(Read more about the query builder.)

With the query builder you don’t have to memorize all the fields the app is sending to Honeycomb. This evens the playing field among team members.

Molly described to her coworkers the mindset she has when investigating interesting data in Honeycomb: “Sometimes I’m not even quite sure how I’m gonna filter down into what I want, but I just start typing and I start scrolling and I start looking.”

BubbleUp is another unique feature in Honeycomb that Molly finds super helpful. “Let me tell you how many times I’ve been on other platforms where all I want is to just narrow down that graph with a drag and drop, and you can’t.”

Rather than trying to correlate system behavior and line up time periods across separate graphs, BubbleUp does the work of comparing the baseline to the outlier data you’ve selected.

Shared History

Another Honeycomb feature Molly loves is the saved queries. “Usually I’m opening like, tab after tab after tab after tab trying to save all my queries. I go down one path and it doesn’t return anything, so then I have to go all the way back.”

Honeycomb automatically saves your query history and displays them in the sidebar. “Having those saved queries, you can literally just go over to your sidebar, you can easily go back, you can go down a rabbit hole and come back to your starting point in seconds. That’s one feature that’s really stood out to me in Honeycomb that I haven’t really seen anywhere else.”

The shared history and query permalinks make it easy to collaborate and get a second set of eyes, or to prevent multiple people from hitting the same dead ends. “If you’re having an incident you can slap the link to people. They can see your history as well so then they can recreate what you did and possibly find issues with it, possibly take it down their own path.”

Level Up Your Team

If you're a Rails developer, you can add the Beeline gem to your app and start sending data today. Go to honeycomb.io/signup to start for free.

What does service ownership mean to you?

shelby spees (she/her) — Wed, 13 May 2020 19:21:44 +0000

What does it mean to have ownership of a software service?

Forem: shelby spees (she/her)

How to set up Twitter cards for your Hugo site

What are social cards?

Twitter cards

Hugo Twitter cards

Hugo layouts

Creating a Twitter card

A User Journey: Setting Up the Node Beeline on Lambda

Getting started

Working with traces and AWS Lambda

HoneyByte: Get a Taste for Sampling

What is sampling?

How does sampling work with Honeycomb?

Using a sample rate

Overriding the sample hook

DEV sampler walk-through

A template custom sample hook

Common sampling pitfalls

How will you sample events?

Interview with Honeycomb Engineer Chris Toshok: Dogfooding OpenTelemetry

Interview with Chris Toshok

Observing the impact

Try it yourself!

Get on the Right Track with Our Rails Integration!

Create a free Honeycomb account

Get your API key

Configure the Ruby Beeline SDK

Generate Your First Events

Get Ready to Ship

Observe Production Traffic

Add Custom Instrumentation

Continue Your Journey!

What's your favorite incident story?

Have you ever been on-call? What was it like?

Observability 101: Terminology and Concepts

Observability Fundamentals

Software Telemetry

Metrics

Logs

Traces

Structured Events

Telemetry concepts

Instrumentation with Honeycomb

Instrumenting code for structured data

What observability tooling needs

Further reading

Reach out!

HoneyByte: Incremental Instrumentation Beyond the Beeline

Liz Fong-Jones

Catch All the Errors!

Compare with Traditional Metrics

Your Turn!

HoneyByte: Make a Beeline Toward Observability Just Like DEV’s Molly Struve

Molly Struve (she/her)

Service Fatigue

Getting Up and Running

Using the Tool

Shared History

Level Up Your Team

What does service ownership mean to you?