Forem: Lightstep

Tracetest in Action: Running Trace-Based Tests on the OpenTelemetry Demo App with Nomad

Adriana Villela — Tue, 28 Feb 2023 15:00:00 +0000

The concept of Trace-based testing (TBT) was first presented by my awesome colleague, Ted Young, at KubeCon North America 2018. The idea behind it was simple, yet elegant: if you’re already creating distributed Traces in your application code, why not use this same Trace data to create test assertions to validate your end-to-end system flow?

Back in 2018, Trace-based testing was just an idea. Fast-forward to today: TBT is now a reality, thanks to Trace standardization à la OpenTelemetry (OTel) and Trace-based testing tools like Tracetest, Helios, and Malabi.

I first heard about TBT in the summer of 2022, when I was writing a piece on how Observability is useful for QAs. After writing the piece, I wanted to get my hands dirty and try this Trace-based testing thing for myself. The tool that I chose for my initial TBT explorations was Tracetest, which is open-source.

Tracetest Overview

Tracetest came out in May 2022, and I first got my hands on Tracetest back in June 2022, so it was still pretty fresh! At the time, it integrated only with Jaeger. More specifically, it used the Jaeger API to pull OTel Traces from Jaeger to register them into Tracetest. You then used the UI to create trace-based tests.

Tracetest has come a long way since those early days! It now has:

A CLI
A quickstart installer so that you can get a full-fledged example up and running on an existing Kubernetes cluster (Very slick and easy to get going, I might add!)
Integrations with a number of Observability back-ends and other tools, including Lightstep.
A declarative approach to defining Trace-based tests via YAML

I got to play around with these newer features last December, after a months-long hiatus, and it was really cool to see the evolution of the product. If you follow my work, you’ll know that I play in both the Kubernetes and Nomad worlds. Today, I’ll be taking you on a quick little guided tour of Tracetest, using Traces from the OpenTelemetry Demo App to give you a feel for how it works. The whole setup will be running on HashiCorp Nomad. \

Tutorial

In this tutorial, I will show you how to install Tracetest (v0.9.1) and the OTel Demo App (v1.1.0) on Nomad running locally using HashiQube. I will then show you how to create and run a simple test with Tracetest, using a couple of Traces from the OTel Demo App.

Although the versions of both Tracetest and the OTel Demo App used in this tutorial are slightly older than what’s out there, you should still get a decent idea of how everything works. The OTel Demo App is configured to send Traces and Metrics to Lightstep via the OTel Collector. The Collector is also configured to send Traces to Tracetest. This configuration is based on guidance from the official Tracetest docs. You can see the full Collector configuration used in this example here.

Please note that there are no official Nomad jobspecs for either Tracetest or the OTel Demo App, so I went ahead and did the conversions myself from Kubernetes manifests to Nomad jobspecs. You can check out the jobspecs in this repo. If you’re curious as to how I went about the Kubernetes-to-Nomad conversion, you can check out my blog post on this topic. I also have a blog post dedicated to running the OTel Demo App on Nomad. If this tickles your fancy, you can check it out here.

Assumptions

Before we move on, I am assuming that you have a basic understanding of:

Nomad. If not, mosey on over to my Nomad intro post. This blog post by Daniela Baron is also great.
Observability (o11y) and OpenTelemetry (OTel). If not, mosey on over to my Observability & OTel post.

Pre-Requisites

In order to run the example in this tutorial, you’ll need the following:

Vagrant (version 2.3.1 at the time of this writing) - to provision HashiQube
Docker (version 20.10.21 at the time of this writing) - we’ll be running HashiQube on Docker, using the Vagrant Docker Provider.

You’ll also need to make sure that your Docker Desktop resource settings are set to the following:

CPUs: 3
Memory: 9.5GB
Swap: 3GB

Tutorial Repos

Below are the repos that we’ll be using for today’s tutorial:

My modified HashiQube Repo (fork of servian/hashiqube). If you’re curious, you can see what modifications I’ve made here.
My Nomad Conversions repo

Part 1: Provision Nomad, Consul, and Vault using HashiQube

In this section, we’ll be provisioning a local Hashi environment (including Nomad, Consul, and Vault) instance using HashiQube. Then, we’ll install Tracetest and the OTel Demo App on Nomad. Feel free to skip this section if you already have a working Hashi environment with Nomad, Consul, and Vault.

1- Update /etc/hosts

We use the Traefik load-balancer to expose our services, which we access as subdomains of localhost. In order ensure that we can access our Traefik-exposed services (and also the Traefik dashboard itself, you’ll need to add the following entries to /etc/hosts on your host machine:

127.0.0.1 traefik.localhost
127.0.0.1 otel-demo.localhost
127.0.0.1 tracetest.localhost

For more on Traefik load-balancing on Nomad, check out this post.

2- Provision a Local Hashi Environment with HashiQube

Start HashiQube by following the detailed instructions here.

You can now access the apps below:

Part 2: Deploying Tracetest and the OTel Demo App on Nomad

We’re ready to deploy the OTel Demo App and Tracetest on Nomad!

1- Add Lightstep Access Token to Vault

As I mentioned earlier, we’ll be sending our Traces to both Lightstep and Tracetest. In order to send Traces to Lightstep, you’ll need to:

Get a Lightstep Access Token. (Make sure that you sign up for a Lightstep account first, if you don’t already have one.)
Configure Vault by following the instructions here.
Add your Lightstep Access Token to Vault by running the command:

vault kv put kv/otel/o11y/lightstep ls_token="<LS_TOKEN>"

Where <LS_TOKEN> is your Lightstep Access Token.

2- Deploy the OTel Demo App and Tracetest

First, let’s clone the repo, and go to our working directory:

git clone https://github.com/avillela/nomad-conversions.git
cd nomad-conversions

Next, let’s deploy the services:

# Enable memory over-subscription
nomad operator scheduler set-config -memory-oversubscription true
# Tracetest deploy
nomad job run -detach tracetest/jobspec/traefik.nomad
nomad job run -detach tracetest/jobspec/postgres.nomad
nomad job run -detach tracetest/jobspec/tracetest.nomad
nomad job run -detach tracetest/jobspec/otel-collector.nomad
nomad job run -detach tracetest/jobspec/go-server.nomad
# OTel Demo App deploy
nomad job run -detach otel-demo-app/jobspec/redis.nomad
nomad job run -detach otel-demo-app/jobspec/ffspostgres.nomad

nomad job run -detach otel-demo-app/jobspec/adservice.nomad
nomad job run -detach otel-demo-app/jobspec/cartservice.nomad
nomad job run -detach otel-demo-app/jobspec/currencyservice.nomad
nomad job run -detach otel-demo-app/jobspec/emailservice.nomad
nomad job run -detach otel-demo-app/jobspec/featureflagservice.nomad
nomad job run -detach otel-demo-app/jobspec/paymentservice.nomad
nomad job run -detach otel-demo-app/jobspec/productcatalogservice.nomad
nomad job run -detach otel-demo-app/jobspec/quoteservice.nomad
nomad job run -detach otel-demo-app/jobspec/shippingservice.nomad
nomad job run -detach otel-demo-app/jobspec/checkoutservice.nomad
nomad job run -detach otel-demo-app/jobspec/recommendationservice.nomad
nomad job run -detach otel-demo-app/jobspec/frontend.nomad
nomad job run -detach otel-demo-app/jobspec/loadgenerator.nomad
nomad job run -detach otel-demo-app/jobspec/frontendproxy.nomad
nomad job run -detach otel-demo-app/jobspec/grafana.nomad
nomad job run -detach otel-demo-app/jobspec/jaeger.nomad
nomad job run -detach otel-demo-app/jobspec/prometheus.nomad

NOTE: We’ve enabled memory oversubscription in Nomad. This is a one-time setting.

Since we’re running the jobs in detached mode, Nomad won’t wait to start the next job until the current one has deployed successfully. This means that your output will look something like this:

Job registration successful
Evaluation ID: d3eaa396-954e-241f-148d-6720c35f34bf
Job registration successful
Evaluation ID: 6bba875d-f415-36b7-bfeb-2ca4b9982acb
Job registration successful
Evaluation ID: 16dc8ef8-5e26-68f4-89b6-3d96b348775b
Job registration successful
Evaluation ID: 34de0532-a3b5-8691-bf18-51c0cc030573
Job registration successful
Evaluation ID: 7310e6a2-9945-710b-1505-c01bd58ccd35
...

Friendly reminder that the Evaluation ID values will be different on your machine.

3- See it in Nomad

As things are deploying, you can mosey on over to the Nomad UI at http://localhost:4646 to see how things are coming along:

It will take some time for all of the services to come up (sometimes up to 30 minutes), depending on network speed (especially since Nomad needs to download the images and initialize the services) and system resources, so be patient! Since some services depend on other services in order to run, you may see services in limbo or some going up and down for a while, per the above screen capture. DON’T PANIC! IT WILL ALL BE OKAY!!

Once all of the jobs are up and running, you’ll see everything look green, like this:

You can also head on over to Consul at http://localhost:8500 to see the health of the services:

By default, unhealthy services show up at the top, with a red “x” next to them. Since we don’t see any nasty red “x”s in the above screen shot, we know that our services are lookin’ good!

4- Access the OTel Demo App

To make sure that the OTel Demo App is up and running, you can access it at: http://otel-demo.localhost.

5- Access Tracetest

You should also be able to access the Tracetest UI at: http://tracetest.localhost

Part 3: Creating & Running Tests in Tracetest

Now that we have the OTel Demo App and Tracetest up and running, we can finally create our tests in Tracetest!

1- Install the Tracetest CLI

The Tracetest CLI allows you to interact with Tracetest programmatically. You can install the Tracetest CLI on Mac via Homebrew:

brew install kubeshop/tracetest/tracetest

For Windows and Linux, check out the instructions here.

Let’s make sure that the CLI is installed properly:

tracetest version

Sample output:

v0.9.1

2- Configure the Tracetest CLI

Now that the CLI is installed, let’s configure the Tracetest CLI so that it knows what Tracetest installation it needs to talk to. Make sure that you’re still in the nomad-conversions root directory, and then run:

cd tracetest
tracetest configure

This will create a file called config.yaml in the directory from which you ran tracetest configure.

When you run the above command, you will be prompted for the Tracetest server URL:

Enter your Tracetest server URL [http://tracetest.localhost]:

Notice that it’s already pre-populated with http://tracetest.localhost. This is because there’s already a config.yaml file in our tracetest directory. If you don’t type anything in and hit return, the CLI will just use the value already in config.yaml. If you want to use a different URL, make sure that you prefix that URL with http:// (or https:// if you’re hosting a secure instance).

Next, you’ll be prompted on whether or not you wish to enable analytics:

Enable analytics? [Y/n]:

If you hit return here, it will enable analytics by default. Type n to disable analytics.

Now, let’s open up config.yaml:

cat config.yaml

Sample output:

scheme: http
endpoint: tracetest.localhost
analyticsEnabled: false

3- Create a Trace-based test in Tracetest

Now we’re ready to create a test! We can do this in one of two ways:

Using the UI (check out the Tracetest docs here for details)
From a YAML file

I’m a big fan of doing things programmatically, so let’s create our test from a YAML file. Per the Tracetest docs, you can create tests from HTTP requests, gRPC requests, cURL commands, and more. In our case, we’re creating our tests from a gRPC request, so our Tracetest test definition file YAML looks like this:

type: Test
spec:
 id: <test_id>
 name: <test_name>
 description: <test_description>
 trigger:
   type: grpc
   grpc:
     protobufFile: |
        <protobuf_file_contents_here>
     address: <grpc_endpoint_address>
     method: <grpc_method_name>
 specs:
 - selector: span[tracetest.span.type="general" name="<operation_name>"]
   assertions:
   - attr:<attribute_name> <operator> <value>

Where:

<test_id> is a unique test identifier. If you create your test via the UI, a unique ID is assigned. If you create a test from a YAML file, you can assign your own ID. For this example, let’s name our ID tt-recsvc-01. As long as it’s unique, you should be good.
<test_name> is the name of your test. Use something short and meaningful.
<test_description> is short description of your test.
trigger.type is grpc, since we’re creating a test from a gRPC request.
trigger.grpc.protobufFile should be the complete contents of your protobuf file. In our case, we’re going to embed the contents of OTel Demo App’s demo.proto file in this field.
<grpc_endpoint_address> is the address at which the gRPC endpoint is served. In this case, we’re using Consul DNS to expose the service’s address. If we look at the Recommendation Service’s Nomad jobspec, you’ll see that the name of the gRPC service is recommendationservice. So when we query it in Consul, it should be accessible at this address recommendationservice.service.consul. We can test this by logging into the HashiQube image. Do this by going to the root directory of HashiQube repo, and typing vagrant ssh. Once you’re in the HashiQube Vagrant box, run this command:

dig +short recommendationservice.service.consul

Expected result:

172.17.0.2

We get the port number from the Recommendation Service jobspec’s port definition stanza. In our case, the port number is 9001. Note that we need to use a static port number in order for this setup to work.

<grpc_method_name> is the name of the gRPC method that we’re testing. It follows the naming convention <package_name>.<service_name>.<method_name>. Where <package_name> is hipstershop, <service_name> is RecommendationService, and <method_name> is ListRecommendations.
spec is where we define our test assertions. We need to first select our target Span(s), and then we define what assertions to apply to the Span(s). In our case, we are selecting a span named get_product_list, and we’re asserting that the value of the attribute app.filtered_products.count is set to 9.

With all that in mind, our YAML should now look like this:

type: Test
spec:
id: tt-recsvc-01
name: RecommendationService test
description: Sample test file for the OTel Demo App RecommendationService
trigger:
  type: grpc
  grpc:
    protobufFile: |
      // Copyright 2020 Google LLC
      //
      // Licensed under the Apache License, Version 2.0 (the "License");
      // you may not use this file except in compliance with the License.
      // You may obtain a copy of the License at
      //
      //      http://www.apache.org/licenses/LICENSE-2.0
      //
      // Unless required by applicable law or agreed to in writing, software
      // distributed under the License is distributed on an "AS IS" BASIS,
      // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
      // See the License for the specific language governing permissions and
      // limitations under the License.

      syntax = "proto3";

      import "google/protobuf/timestamp.proto";

      package hipstershop;

      option go_package = "genproto/hipstershop";

      // -----------------Cart service-----------------

      service CartService {
          rpc AddItem(AddItemRequest) returns (Empty) {}
          rpc GetCart(GetCartRequest) returns (Cart) {}
          rpc EmptyCart(EmptyCartRequest) returns (Empty) {}
      }

      message CartItem {
          string product_id = 1;
          int32  quantity = 2;
      }

      message AddItemRequest {
          string user_id = 1;
          CartItem item = 2;
      }

      message EmptyCartRequest {
          string user_id = 1;
      }

      message GetCartRequest {
          string user_id = 1;
      }

      message Cart {
          string user_id = 1;
          repeated CartItem items = 2;
      }

      message Empty {}

      // ---------------Recommendation service----------

      service RecommendationService {
        rpc ListRecommendations(ListRecommendationsRequest) returns (ListRecommendationsResponse){}
      }

      message ListRecommendationsRequest {
          string user_id = 1;
          repeated string product_ids = 2;
      }

      message ListRecommendationsResponse {
          repeated string product_ids = 1;
      }

      // ---------------Product Catalog----------------

      service ProductCatalogService {
          rpc ListProducts(Empty) returns (ListProductsResponse) {}
          rpc GetProduct(GetProductRequest) returns (Product) {}
          rpc SearchProducts(SearchProductsRequest) returns (SearchProductsResponse) {}
      }

      message Product {
          string id = 1;
          string name = 2;
          string description = 3;
          string picture = 4;
          Money price_usd = 5;

          // Categories such as "clothing" or "kitchen" that can be used to look up
          // other related products.
          repeated string categories = 6;
      }

      message ListProductsResponse {
          repeated Product products = 1;
      }

      message GetProductRequest {
          string id = 1;
      }

      message SearchProductsRequest {
          string query = 1;
      }

      message SearchProductsResponse {
          repeated Product results = 1;
      }

      // ---------------Shipping Service----------

      service ShippingService {
          rpc GetQuote(GetQuoteRequest) returns (GetQuoteResponse) {}
          rpc ShipOrder(ShipOrderRequest) returns (ShipOrderResponse) {}
      }

      message GetQuoteRequest {
          Address address = 1;
          repeated CartItem items = 2;
      }

      message GetQuoteResponse {
          Money cost_usd = 1;
      }

      message ShipOrderRequest {
          Address address = 1;
          repeated CartItem items = 2;
      }

      message ShipOrderResponse {
          string tracking_id = 1;
      }

      message Address {
          string street_address = 1;
          string city = 2;
          string state = 3;
          string country = 4;
          string zip_code = 5;
      }

      // -----------------Currency service-----------------

      service CurrencyService {
          rpc GetSupportedCurrencies(Empty) returns (GetSupportedCurrenciesResponse) {}
          rpc Convert(CurrencyConversionRequest) returns (Money) {}
      }

      // Represents an amount of money with its currency type.
      message Money {
          // The 3-letter currency code defined in ISO 4217.
          string currency_code = 1;

          // The whole units of the amount.
          // For example if `currencyCode` is `"USD"`, then 1 unit is one US dollar.
          int64 units = 2;

          // Number of nano (10^-9) units of the amount.
          // The value must be between -999,999,999 and +999,999,999 inclusive.
          // If `units` is positive, `nanos` must be positive or zero.
          // If `units` is zero, `nanos` can be positive, zero, or negative.
          // If `units` is negative, `nanos` must be negative or zero.
          // For example $-1.75 is represented as `units`=-1 and `nanos`=-750,000,000.
          int32 nanos = 3;
      }

      message GetSupportedCurrenciesResponse {
          // The 3-letter currency code defined in ISO 4217.
          repeated string currency_codes = 1;
      }

      message CurrencyConversionRequest {
          Money from = 1;

          // The 3-letter currency code defined in ISO 4217.
          string to_code = 2;
      }

      // -------------Payment service-----------------

      service PaymentService {
          rpc Charge(ChargeRequest) returns (ChargeResponse) {}
      }

      message CreditCardInfo {
          string credit_card_number = 1;
          int32 credit_card_cvv = 2;
          int32 credit_card_expiration_year = 3;
          int32 credit_card_expiration_month = 4;
      }

      message ChargeRequest {
          Money amount = 1;
          CreditCardInfo credit_card = 2;
      }

      message ChargeResponse {
          string transaction_id = 1;
      }

      // -------------Email service-----------------

      service EmailService {
          rpc SendOrderConfirmation(SendOrderConfirmationRequest) returns (Empty) {}
      }

      message OrderItem {
          CartItem item = 1;
          Money cost = 2;
      }

      message OrderResult {
          string   order_id = 1;
          string   shipping_tracking_id = 2;
          Money shipping_cost = 3;
          Address  shipping_address = 4;
          repeated OrderItem items = 5;
      }

      message SendOrderConfirmationRequest {
          string email = 1;
          OrderResult order = 2;
      }


      // -------------Checkout service-----------------

      service CheckoutService {
          rpc PlaceOrder(PlaceOrderRequest) returns (PlaceOrderResponse) {}
      }

      message PlaceOrderRequest {
          string user_id = 1;
          string user_currency = 2;

          Address address = 3;
          string email = 5;
          CreditCardInfo credit_card = 6;
      }

      message PlaceOrderResponse {
          OrderResult order = 1;
      }

      // ------------Ad service------------------

      service AdService {
          rpc GetAds(AdRequest) returns (AdResponse) {}
      }

      message AdRequest {
          // List of important key words from the current page describing the context.
          repeated string context_keys = 1;
      }

      message AdResponse {
          repeated Ad ads = 1;
      }

      message Ad {
          // url to redirect to when an ad is clicked.
          string redirect_url = 1;

          // short advertisement text to display.
          string text = 2;
      }

      // ------------Feature flag service------------------

      service FeatureFlagService {
        rpc GetFlag(GetFlagRequest) returns (GetFlagResponse) {}
        rpc CreateFlag(CreateFlagRequest) returns (CreateFlagResponse) {}
        rpc UpdateFlag(UpdateFlagRequest) returns (UpdateFlagResponse) {}
        rpc ListFlags(ListFlagsRequest) returns (ListFlagsResponse) {}
        rpc DeleteFlag(DeleteFlagRequest) returns (DeleteFlagResponse) {}
      }

      message Flag {
        string name = 1;
        string description = 2;
        bool enabled = 3;
        google.protobuf.Timestamp created_at = 4;
        google.protobuf.Timestamp updated_at = 5;
      }

      message GetFlagRequest {
        string name = 1;
      }

      message GetFlagResponse {
        Flag flag = 1;
      }

      message CreateFlagRequest {
        string name = 1;
        string description = 2;
        bool enabled = 3;
      }

      message CreateFlagResponse {
        Flag flag = 1;
      }

      message UpdateFlagRequest {
        string name = 1;
        bool enabled = 2;
      }

      message UpdateFlagResponse {}

      message ListFlagsRequest {}

      message ListFlagsResponse {
        repeated Flag flag = 1;
      }

      message DeleteFlagRequest {
        string name = 1;
      }

      message DeleteFlagResponse {}
    address: recommendationservice.service.consul:9001
    method: hipstershop.RecommendationService.ListRecommendations
specs:
 - selector: span[name="get_product_list"]
   assertions:
   - attr:app.filtered_products.count  =  9

Save this file as recsvc-test.yml in the tracetest/tests directory of nomad-conversions.

4- Run the Trace-based test

With our test YAML in hand, we can run our test! Make sure that you’re running this from tracetest directory of nomad-conversions:

tracetest test run --definition tests/recsvc-test.yml

Sample output:

✔ RecommendationService test (http://tracetest.localhost/test/avtest4567/run/1/test)

Now let’s check out our test in the Tracetest UI by going to http://tracetest.localhost:

Let’s click on the > next to the test definition to expand it:

Note that the UI shows that it ran v1 of the test. If we make any changes to the test definition, Tracetest bumps the test version; however, if we were to change only the test ID and re-run the file, it would show up as an entirely different test in Tracetest, and not a different version of the same test.

Let’s explore our test definition in the UI, by clicking in the area highlighted below:

Will take us to the test trigger definition screen:

This might seem familiar - it’s the information that we entered in our test YAML. Tracetest runs the API endpoint and then checks to see if there are any Traces associated with this with this endpoint, and lets you create tests against the traces. Test specs are defined in the Test tab, located at the top middle section of the screen:

Clicking on it brings us to the test spec definition screen:

Next, let’s head to the top left-hand side of the screen, and click on the purple circle with >>:

This reveals a side pane with all sorts of information about the selected Span. In our case, we’ve selected the Tracetest trigger Span (top box), so our screen looks something like this:

Notice how the Tracetest trigger box has a slight blue highlight. But we’re not interested in this Span. We’re actually interested in the get_product_list Span, so let’s click on that Span:

Notice how it has an attribute called app.filtered_products.count, and that its value is 9…which is the value that we’re checking for in our assertion! Ta-da! 🎉

Part 4: See it Lightstep!

Remember that our OTel Collector is configured to send Traces to both Tracetest and Lightstep, so we should see the same Trace data in both.

I went into my Lightstep account and created a Notebook that looked like this:

And clicked on one of the matching traces returned, which led me to the Trace diagram below:

As you can see, we have our get_product_list trace, which has an attribute called app.filtered_products.count with a value of 9. Yay!

Final Thoughts

Tracetest is a great tool for unlocking the potential of Trace-based testing. Although the example in this tutorial was fairly simple, I hope that it gives you an appreciation for some of the cool stuff that Tracetest can do. I definitely plan to keep my eye out as this tool improves, and I am excited to see where Trace-based testing takes us in 2023!

For more on the ins and outs of Tracetest, be sure to check out the official docs here. For you visual learners, I’ve put together a quick little four-minute video highlighting the main things covered in this post. Be sure to check it out here.

That was definitely a lot to take in, so give yourself a pat on the back for getting through this! Now please enjoy this lovely photo of Mookie the rat being cradled in my husband’s arms.

Peace, love, and code!

This post was originally published on the Tracetest blog.

Three Terraform Mistakes, and How to Avoid Them

Adriana Villela — Mon, 21 Nov 2022 15:56:17 +0000

In my last blog post, I talked about how Ana Margarita Medina and I used Terraform to show off Observability-Landscape-as-Code in practice, leveraging the OpenTelemetry Demo App to do so. The Demo App showcases instrumentation of Traces and Metrics of different services written in different languages using OpenTelemetry (OTel). Our Terraform code did the following:

Created a Kubernetes cluster
Deployed the Demo App to Kubernetes
Deployed OpenTelemetry Collector to Kubernetes, and configured it to send Traces and Metrics to Lightstep
Created dashboards in Lightstep

Now, I’m a fan of beautiful code, so we organized our code using Terraform Modules. We used a module for provisioning the Kubernetes cluster, one for deploying the OTel Demo App and the OTel Collector, and one for creating the Lightstep dashboards.

We also leveraged the following Terraform Providers:

Google Cloud Provider for spinning up a Kubernetes Cluster in Google Cloud Platform (GCP)
The Kubernetes Provider and the Helm Provider for deploying the app to Kubernetes
The Lightstep Provider for creating the dashboards in Lightstep

All good, right? Except for one teeeensy little problem...the last time I’d touched Terraform was in early 2021, and even then, I was just tweaking code. So I kinda had to teach myself Terraform all over again. And I hit up a few snags along the way. Cue. The. Panic.

Fortunately, Google came through, and we were able to resolve the issues. In today’s blog post, I will cover THREE Terraform gotchas that Ana and I hit, and how we solved them, so that you will hopefully be spared our utter despair and panic. 😅

Let’s do this!

NOTE: If you want to follow along to see the full Terraform source code, you can check it out here. Even though the source code is specific to the Observability-Landscape-as-Code use case, the main Terraform concepts in this blog post can be ported over to other scenarios.

Gotcha #1: The Chicken-and-Egg Scenario

After creating a Kubernetes cluster, we needed to create a Kubernetes resource before we could apply the Helm chart to install the OpenTelemetry demo app. The Demo App’s Helm Chart deploys an OpenTelemetry Collector. We wanted to configure the Collector to send OTel data to Lightstep. To do so, you need to add a Lightstep Access Token, which is stored as a Kubernetes secret.

You can learn more about the specifics of this setup here.

To create the secret in Kubernetes before running the Helm Chart, we used the Kubernetes Provider. In order to use this provider, Terraform needs to know information about your cluster, so that it knows what cluster to apply the manifest to. To do this, I needed to store the cluster information in the data stanza, like this:

data "google_client_config" "default" {}

data "google_container_cluster" "primary" {
 name     = var.cluster_name
 location = var.region
}

provider "kubernetes" {
 host  = "https://${data.google_container_cluster.primary.endpoint}"
 token = data.google_client_config.default.access_token
 cluster_ca_certificate = base64decode(
   data.google_container_cluster.primary.master_auth.0.cluster_ca_certificate
 )
}

Easy peasey, right? Unfortunately, when I ran terraform apply, I kept getting the following errors:

Error: Invalid template interpolation value

And

Error: Attempt to index null value

Basically, Terraform was trying to evaluate the contents of the data stanza (which were null) before it had any information about the Kubernetes cluster. Which of course it didn’t, because the cluster didn’t yet exist!! Hence the null contents.

I frantically Googled this one for a while, spinning my wheels. And then, the “aha” moment hit me, when I saw somewhere in one of my searches that I could use the depends_on attribute in the data stanza. So I added depends_on = [module.k8s_cluster_create] to both my data stanzas, which basically says, “Hey buddy, don’t try to evaluate this until AFTER the k8s_cluster_create module (i.e. the module in which the Kubernetes cluster is created) is run. So now, after adding depends_on, my providers.tf (lines 32-49) code looked like this:

data "google_client_config" "default" {
 depends_on = [module.k8s_cluster_create]
}

data "google_container_cluster" "primary" {
 depends_on = [module.k8s_cluster_create]
 name     = var.cluster_name
 location = var.region
}

provider "kubernetes" {
 host  = "https://${data.google_container_cluster.primary.endpoint}"
 token = data.google_client_config.default.access_token
 cluster_ca_certificate = base64decode(
   data.google_container_cluster.primary.master_auth.0.cluster_ca_certificate
 )
}

And after making that change, all was well with the world. Huzzah!

Gotcha #2: Using Modules with depends_on

While the above problem went away, I then found myself face-to-face with yet another conundrum. When I initially wrote my Terraform code, everything was in one big file, and it worked just fine. So OF COURSE I just assumed that when I prettified my code and moved things into modules, I could just get away defining my Providers in the Modules themselves. Well, you can. That is…if you don’t use the depends_on attribute in your Module call.

So basically, when I tried to say that the Module lightstep_dashboards depended on k8s_cluster_create like this:

module "k8s_cluster_create" {
   source = "./modules/k8s"

   cluster_name = var.cluster_name
   project_id = var.project_id
   region = var.region
   network = var.network
   subnet = var.subnet
}

module "deploy_otel_demo_app" {
   source = "./modules/otel_demo_app"

   otel_demo_namespace = var.otel_demo_namespace
   ls_access_token = var.ls_access_token
   cluster_name = var.cluster_name
   project_id = var.project_id
   region = var.region
   network = var.network
   subnet = var.subnet
}

module "lightstep_dashboards" {
   source = "./modules/lightstep"
   depends_on = [module.k8s_cluster_create]

   lightstep_project = var.ls_project
}

I kept getting this error when I ran terraform apply:

Error: Module is incompatible with count, for_each and depends_on

This error happens when the Child Module contains a provider block and the Module that you’re trying to call is using count, depends_on, and/or for_each. Why? Because provider blocks inside a Child Module are not allowed when your Module call is using count, depends_on, and/or for_each. You can read up more on this here.

Well, it turns out that correct practice is to define your provider block in the Root Module, as Providers are automagically passed down to the Child Modules. So to make the above error go away, I moved all of my Provider definitions to the Root Module, and was able to keep depends_on in my Module call. If I didn’t have any dependencies, I could’ve left out the depends_on block, but I wouldn’t really be following the recommended practice.

NOTE: You can learn more about Providers and Modules here.

Gotcha #3: Referencing a non-TF provider in a module

Two problems down. Awesome! Unfortunately, my problems were not over. I continued to anger the Module Gods, because I encountered yet another issue when I moved my non-modularized code into Modules. This time, it had to do with using the Lightstep Provider. You see, this Provider comes from a third-party (i.e. not HashiCorp), which in this case is Lightstep. Lightstep is what is known as a Partner Provider. This means that in the Provider Registry, the Provider is named lightstep/lightstep, where the first lightstep means that the Provider is created and maintained by Lightstep, and the second lightstep is the actual Provider name. For comparison, the hashicorp/google provider is an Official Provider, because it is created and maintained by HashiCorp.

Now here’s the odd part. When I tried to run terraform init, I was graced with this error:

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/lightstep

Um...what? This did not compute, because in my providers.tf file, I CLEARLY said that the Provider name was lightstep/lightstep, so where oh where was it getting this hashicorp/lightstep business from?? LOOK ⬇️⬇️⬇️

terraform {
 required_providers {
...
   lightstep = {
     source = "lightstep/lightstep"
     version = ">=1.70.0"
   }
...
}

O Google gods, help meeeeee!!

Well, it turns out that when using a Partner Provider in a Module, Terraform assumes the Provider is an Official Provider, and is therefore automagically given a hashicorp suffix when passing it down to the Module. So Terraform basically thought that the Provider was called hashicorp/lightstep, even though I clearly defined it correctly in the Providers section of the Root Module.

To fix this issue, I ended up having to define a required_providers stanza in the Root Module, as I had already done, AND I also had to add a required_providers stanza to my Child Module, as per the snippet below:

terraform {
 required_providers {
   lightstep = {
     source = "lightstep/lightstep"
     version = ">=1.70.0"
   }
 }
}

After that, my terraform init stopped screaming at me!

Final Thoughts

Today we learned that Terraform can be a wee finicky. We learned that:

Adding depends_on to the data stanza used to capture your Kubernetes cluster configuration data ensures that Terraform doesn’t try to evaluate the data stanza until AFTER the cluster is created, thereby avoiding some serious Terraform Anger™.
If you want to use depends_on in a Module call, the Provider configuration must be done in the Root Module. Also, it’s the recommended practice even if you don’t want to use depends_on.
If you have a Module that references a Partner Provider, you need to define a required_providers stanza in both the Root Module and the Child Module.

I hope that these tips prevent you from experiencing Terraform Anguish™ next time you find yourself Terraformin’. And now, I shall reward you with a picture of my rat Mookie, who is seen below peering out of an authentic Wisconsin Cheese Head hat.

Peace, love, and code. 🦄 🌈 💫

Got questions about Terraform or Observability-Landscape-as-Code? Talk to me! Feel free to connect through e-mail, or hit me up on Twitter, Mastodon, or LinkedIn. Hope to hear from y’all!

Observability-Landscape-as-Code in Practice

Adriana Villela — Tue, 15 Nov 2022 15:47:02 +0000

with Ana Margarita Medina

If you follow Adriana's writings on Observability, you may recall a post from back in June introducing the concept of Observability-Landscape-as-Code (OLaC).

An Observability Landscape is made up of the following pieces:

Application instrumentation
Collecting and storing application telemetry
An Observability back-end
A set of meaningful SLOs
Alerts for on-call Engineers

Keeping that in mind, OLaC is simply the codification of your Observability Landscape, thereby ensuring consistency, maintainability, and reproducibility.

That’s all well and good, but how about seeing this thing in action? Well, my friend, you’ve come to the right place, because today, you get to see a tutorial featuring a number of OLaC practices in action!

Today, Ana Margarita Medina (my fellow DevRel and partner in crime) and I will be highlighting the following aspects of OLaC:

Application instrumentation

How? À la OpenTelemetry, via the OpenTelemetry Demo App, which contains examples of Traces and Metrics instrumentation.
Collecting & storing application telemetry

How: OpenTelemetry Collector is deployed via code (Helm chart), alongside the various services that make up the OpenTelemetry Demo App.
Codifying your Observability back-end configuration

How: Using the Lightstep Terraform Provider to create dashboards in Lightstep.

We wanted to showcase OLaC principles with a real-life example using modern cloud-native tooling...Which means using Kubernetes for our cloud infrastructure with Google Cloud’s Kubernetes offering, GKE. Now, since we are good practitioners of OLaC and SRE, we won’t just be setting things up through the clickity click of a UI. No sirreee. Instead, we’ll be #automatingAllTheThings using HashiCorp Terraform. Terraform allows us to do infrastructure-as-code (IaC), and gives us tons of added benefits like better control over our resources and standardization. These are key principles in OLaC and IaC.

We will be deploying OpenTelemetry Demo App to our cluster. The Demo App has been instrumented using OpenTelemetry, and will send Traces and Metrics through the OpenTelemetry Collector to Lightstep.

Are you ready??? Let’s get started!

Tutorial

Pre-Requisites

Before you begin, you will need the following:

A Lightstep account so you can see application Traces, and Metrics dashboards
A Lightstep Access Token for the Lightstep project you would like to use
A Lightstep API key for creating dashboards in Lightstep.
Terraform CLI to run the Terraform scripts
A Google Cloud account, so you can create a Kubernetes cluster (GKE)
gcloud CLI to interact with Google Cloud
kubectl to interact with Kubernetes

Steps

In today’s tutorial, we’ll be running Terraform code to showcase OLaC in action. For convenience, the main components have been broken up into separate Terraform modules that will do the following:

Create a Kubernetes cluster (GKE) in Google Cloud using the Google Terraform Provider. This is defined in the k8s module in our repo.
Deploy the OpenTelemetry Demo App to the cluster using the Helm Terraform Provider. This is defined otel_demo_app module in our repo.
Create Metrics dashboards in Lightstep Terraform Provider. This is defined in the lightstep module in our repo.

The full code listing for this tutorial can be found here.

1- Clone the example repo

Let's start by cloning the example repo:

git clone https://github.com/lightstep/unified-observability-k8s-kubecon.git

2- Initialize Sub-Modules

This project makes use of a few Git submodules, so in order to ensure that things work nicely, you’ll need to pull them in:

cd unified-observability-k8s-kubecon
git submodule init && git submodule update

3- Google Cloud Login

Before we can create a GKE cluster you must authenticate your Google Cloud account:

gcloud auth application-default login --no-launch-browser

You will be presented with a link which you need to open up in a browser, to authenticate your Google ID. Once you are authenticated, the browser will display an authorization token for you to paste in the command line, as follows:

4- Create terraform.tfvars

Now that you’re authenticated, let’s get ready to Terraform! Before you can do that, we need to create a terraform.tfvars file.

Lucky for you, we have a handy-dandy template that you can use to get started:

cd k8s-cluster-with-otel-demo/terraform
cp terraform.tfvars.template terraform.tfvars

Next populate the following values in the file:

<your_gcp_project>: The name of your Google Cloud project. Don’t know your project name? No problem! Just run gcloud config get-value project to find out what it is!
<your_gke_cluster_name>: The name you wish to give your GKE cluster. Make sure it follows Kubernetes cluster naming conventions (i.e. no underscores _ or special characters).
<your_lightstep_access_token>: Your Lightstep Access Token. This is used to send Traces to your Lightstep Project.
<your_lightstep_api_key>: Your Lightstep API key. This is used to create our Metrics dashboards.
<your_lightstep_org_name>: Your Lightstep organization name. Not sure what your organization is called? No problem! Log into Lightstep,and click on the person icon on the bottom left of your screen. This will pop up a little menu. The organization name can be found under the “Account Management” heading, like this:

Notice that my organization is called “LightStep”. Yours will be different. Note also that Organization names are case-sensitive.

<your_lightsep_project_name>: The name of the Lightstep Project where the Metrics dashboards will be displayed.

Note: terraform.tfvars is in .gitignore and won't be put into version control.

5- Run Terraform

This step will initialize Terraform (install providers locally), and then will apply the Terraform plan.

It will:

Create a Kubernetes cluster
Deploy the OpenTelemetry Demo App using the OpenTelemetry Demo Helm Chart
Create dashboards in Lightstep

Before running the commands below, make sure that you’re already in the k8s-cluster-with-otel-demo/terraform folder.

terraform init
terraform apply -auto-approve

Please note that this step may take up to 30 minutes, depending on GKE’s disposition. Be patient. 😄

6- Update your kubeconfig

Now that the cluster is created, you can add it to your kubeconfig file! By default, the file is saved at $HOME/.kube/config.

Before you can update your kubeconfig, you first need to make sure that you have the gke-gcloud-auth-plugin installed:

gcloud components install gke-gcloud-auth-plugin
gke-gcloud-auth-plugin --version
echo "export USE_GKE_GCLOUD_AUTH_PLUGIN=True" >> ~/.bashrc

Now we can add the cluster to kubeconfig:

gcloud container clusters get-credentials $(terraform output -raw kubernetes_cluster_name) --region $(terraform output -raw region)

This gets the kubernetes_cluster_name and region output values from Terraform (that’s the terraform output -raw stuff), and plunks those into your gcloud container clusters get-credentials command.

Or, if you closed the terminal in which you were running Terraform and lost your output values, you can also do this:

gcloud container clusters get-credentials <cluster_name> --region <region>

Where <cluster_name> and <region> correspond to the values you entered in Step 3 in your terraform.tfvars file.

7- Check out the OTel Demo app

Now that you’ve got the OpenTelemetry Demo App deployed to your cluster, let’s take a look at it! First, let’s peek into Kubernetes to see what’s up.

If you run kubectl get ns, you’ll notice that there’s now a new namespace called otel-demo:

This is where we deployed the OTel Demo app. Let’s look into this namespace to see what we’ve created. First, let’s look at the pods with kubectl get pods -ns otel-demo:

Notice how we deployed a bunch of different services that make up the OTel Demo App, including adservice, cartservice, recommendationservice, etc.

We also deployed an OTel Collector. Its configuration YAML is stored in a configmap. We can take a peek by running kubectl describe configmap otel-demo-app-otelcol -n otel-demo:

You can see that we also reference a variable called ${LS_TOKEN} which represents your Lightstep Access Token, which you set in terraform.tfvars. But where is it? The secret is mounted to the OTel Collector container instance as a secret called otel-collector-secret. Let’s take a look at the secret by running kubectl describe secret otel-collector-secret -n otel-demo:

All this magic happens in otel-demo-app-values-ls.yaml. This is a version of values.yaml from the OTel Demo App Helm Chart with updates to the Collector configs so that we can configure the OTel Collector to send Traces to Lightstep.

8 - Run the OTel Demo App

Okay...enough Kubernetes talk. Let’s look at the OpenTelemetry Demo App! You can access the Demo App by Kubernetes port-forward:

kubectl port-forward -n otel-demo svc/otel-demo-app-frontend 8080:8080

To access the front-end, go to http://localhost:8080:

Go ahead and explore the amazing selection of telescopes and accessories, and buy a few. 😉🔭

9- See Traces in Lightstep

We can now pop over to Lightstep and check out some Traces. Let’s do this by creating a Notebook.

First, click on the little page icon on the left nav bar (highlighted in blue, below). That will bring up this page:

Next, we build our query for our Traces. Let’s look at the traces from the recommendationservice. We’ll do by entering recommendationservice in the field next to “All telemetry”. Because this is a service, select the second value from the drop-down, which says, “Use ‘recommendationservice’ as service value”, as per below:

After you select that value, you’ll see a chart like this:

The little green dots represent trace exemplars from that Service. Hover over one of them to see for yourself!

If you click on one of these dots, you’ll get taken to the Trace view. Before you click, be sure to save your Notebook first (don’t worry, you’ll get a reminder before you navigate away from the page)!

Here’s the Trace view we see when we click on the get_product_list dot (Operation) above::

Pretty cool, amirite?

10- See Kubernetes Metrics in Lightstep

But wait...there’s more! Not only can we see Traces in Lightstep, we can also see Metrics!

Remember when you ran terraform apply? Well, not only did it create a Kubernetes cluster, deploy the OTel Demo App (and OTel Collector), it also created some handy-dandy Metrics dashboards for us.

You can check out the newly-created Metrics dashboards by going to the Dashboards icon (the icon with 4 little squares) on the left navigation bar:

First, let’s check out the Kubernetes / Compute Resources / Cluster dashboard. This dashboard lets you see the state of your cluster.

We then have various other Metrics called Kubernetes Workload Metrics. These are the dashboards with names that start with “Kubernetes / Compute Resources / Workload”. These dashboards are specific to the services you are running. They take into account the Kubernetes Workloads in your various namespaces, using kube-state-metrics. For a closer look, check out otel_demo_app_k8s_dashboard.tf.

We used Lightstep’s Prometheus Kubernetes OpenTelemetry Collector to get these Metrics into Lightstep. This Helm chart is inspired by kube-prometheus-stack, but with one crucial difference -- no Prometheus! We’re able to use recent enhancements to the OpenTelemetry Operator for Kubernetes such as support for Service Monitors in order to scrape Prometheus metrics from pods, system components, and more.

The best part is that you don’t need to install and maintain a Prometheus instance to be able to run it! But wait…there’s more! You also don’t need to use Lightstep as your Observability back-end in order to take advantage of this special Collector! How cool is that??

Note: You can learn more about the Prometheus Kubernetes OpenTelemetry Collector by checking out the docs here.

For example, the Kubernetes / Compute Resources / Workload / otel-demo-app-cartservice dashboard displays metrics for the OTel Demo App’s cartservice. In it we can see how our containers and pods are doing based on Metrics such as those for CPU and Memory.

11- See Application Metrics in Lightstep

Ah...but we’re not done with Metrics just yet! If you go back to the dashboard view and scroll to the very end of the list, you’ll see the OTel Demo App - Application Metrics dashboard.

Let’s click on it to take a quick little peek!

The latest version of the OTel Demo App emits both auto-instrumented and manually-instrumented Metrics. In today’s demo, we wanted to highlight some of the Metrics from the recommendationservice.

First, we have the auto-instrumented Python Metrics, which are captured from the Python runtime:

runtime.cpython.cpu_time: Track the amount of time being spent in different states of the CPU. This includes user (time running application code) and system (time spent in the operating system). This metric is represented as total elapsed time in seconds.
runtime.cpython.memory: Memory utilization
runtime.cpython.gc_count: Number of times the garbage collector has been called.

We also have one manually-instrumented Metric:

app_recommendations_counter: Cumulative count of the number of recommended products per service call

For more on the recommendationservice Metrics, check out this doc. For more on Metrics captured by other services, check out the OTel Demo App service docs.

12- Teardown

If you’re no longer using this environment, don’t forget to tear down its resources, to avoid running up a huge cloud bill. You’re welcome. 😉

terraform destroy -auto-approve

This step can take up to 30 minutes, so please be patient! Also, you’ll probably notice that on first run, you’ll see the following error:

Don’t panic! If you run terraform destroy -auto-approve again, it will finish nukifying all the things.

Final Thoughts

Today we got to see some aspects of Observability-Landscape-as-Code (OLaC) in practice! Specifically, we looked at the following elements:

Application instrumentation with OpenTelemetry
Collecting and storing application telemetry via the OTel Collector
Configuring an Observability back-end (i.e. Lightstep) through code

We showcased this by using Terraform to:

Deploy the OpenTelemetry Demo App to Kubernetes. The Otel Demo App showcases the Traces and Metrics instrumentation of different services in different languages using OpenTelemetry.
Deploy an OpenTelemetry Collector to Kubernetes (part of the Demo App deployment). The Collector is used to send application Traces and Metrics to Lightstep.
Configure Lightstep dashboards. The Lightstep Terraform provider allowed us to codify this.

Codifying our Observability Landscape means that we can tear down and recreate our application, Collector, and dashboards as needed, knowing that we’ll have consistency across the board every single time. Plus, it means that we can version control it, so that it’s not lost in the ether somewhere, or sitting in a secret server under Bob’s desk. Bonus!

Hopefully this gives you a nice little flavour of the power of OLaC, and will inspire you to go out there and start OLaC-ing too! (I just made up a new verb. You’re welcome.)

Whew! That was a lot to think about and take in! Give yourself a pat on the back, because we’ve covered a LOT! Now, please enjoy this picture of Adriana’s rat, Bunny, enjoying an almond!

Peace, love, and code. 🦄 🌈 💫

The OpenTelemetry Demo App is always looking for feedback and contributors. Please consider joining the OTel Community to help make OpenTelemetry AWESOME!

Got questions about Observability-Landscape-as-Code? Talk to us! Feel free to connect with us through e-mail, or:

Connect with Adriana on Twitter or LinkedIn
Connect with Ana on Twitter or LinkedIn.

Hope to hear from y’all!

KubeCon North America 2022: A Retrospective

Adriana Villela — Wed, 09 Nov 2022 20:15:53 +0000

with Ana Margarita Medina

Hello fellow Kubernetes nerds! 🤓 KubeCon North America is now behind us, so it’s time for a conference retrospective! Now, y’all must be thinking, “Awww, not another one! Noooo...” Ah, but ‘tis not just any KubeCon wrap-up, because both Ana and I will be sharing our perspectives, and we’re awesome and y’all love us. Amirite? Course I am. Without further ado...

Adriana’s Point-of-View

If there is any conference that is on My Conference Bucket List™, it is most definitely KubeCon. I have a love-hate relationship with Kubernetes (don’t we all, though?), and have spent hours trying to understand its wiley ways, and cursing at my terminal at yet another CrashLoopBackOff. So to go to KubeCon and nerd out on Kubernetes just sounded awesome to me.

I finally got my chance by attending KubeCon North America 2022, which took place in Detroit, MI.

I was super excited to attend the conference for a few reasons.

It would be the first time I’d be getting to meet my Lightstep DevRel team, and find out if I was the shortest of the bunch. Spoiler alert: Ana and I are the shortest at around 5’3 (161cm for my Metric peeps). Latina genes. I roll with it.
It was my conference début as a DevRel. Y’all, this is my first DevRel role, if you can believe it - 6 months in and lovin’ it!
Last time I attended a conference was DevOps Days Toronto, in 2018. The highlight of that was getting Dr. Nicole Forsgren and Jez Humble (both of DORA fame) to sign a hard copy of their book, Accelerate, so not too shabby. PS: Get the book. Bonus: if you get the audiobook, it’s read by Nicole.

So, enough rambling. Here’s my KubeCon recap, newbie edition!

As a COVID-cautious gal, I was pleased to see that the CNCF had various COVID precautions in place including vaccine requirement or a negative COVID test to pick your badge and masking was required and enforced. various COVID precautions in place. For starters, you needed proof of vaccination or proof of a negative COVID test in order to pick your badge. Inside the venue, masking was required and enforced. I always mask indoors (no exceptions), so I definitely felt more at ease among the droves of humans, knowing that we had those protections in place. The other thing that I very much appreciated is that the CNCF provided freebie rapid COVID tests proctored through eMed, so I was able to test daily while I was there, for extra peace of mind.

Although KubeCon itself didn’t start until Wednesday, October 26th, I flew in on Sunday night so that I could catch Open Observability Day on Monday, and OTel Unplugged on Tuesday.

Day 1: Open Observability Day

I popped in and out of Open Observability Day, and of the talks that I caught, there were two talks that I really enjoyed. One was OTel Me How to Build a Data Pipeline for Observability, by Daniel Kim and Reese Lee. Even though I was already familiar with the content, I really appreciated Daniel and Reese’s super energetic presentation style. It was definitely a welcome pick-me-up to help fight post-lunch food coma. The other talk I really enjoyed was What Can eBPF Actually do for Modern-Day Observability? by Ori Shussman. In it, he talks about how eBPF lets us see data that is otherwise not visible. For example, eBPF is useful for providing greater insight into gRPC calls, which are notoriously difficult to observe. 🤯I also wanted to give a special shout-out to Opening the Door to Observability, by Libby Meren. Although I missed this talk, this topic is very near and dear to my heart, because in my previous role, I had to spend a chunk of time trying to get buy-in on doing Observability The Right Way ™.

Interested in checking out the other talks? Y’all can check out videos for them here.

Later that afternoon, I got to meet my Lightstep DevRel fam in real life, and it was AWESOME! Unfortunately, we never took a group selfie. 😿

Day 2: OTel Unplugged

I spent Day 2 at the Colony Club to attend OTel Unplugged. This event was sponsored by Lightstep, Honeycomb, New Relic, Splunk, Dynatrace, Crowdstrike, and NGINX. I came into the event not knowing what to expect. I can sometimes clamp up when I’m around folks that I don’t know, but because I was helping with the event check-in, I got to say hello to a number of the attendees, which helped break the ice. And it turns out that there were a lot of names that I recognized from my work in the OTel community, and it was nice to connect in person with folks whom I’d only previously met through Slack or Zoom.

PS: Big shout-out to the venue, which was gorgeous, and the staff were super friendly.

Day 3: KubeCon!

Finally, the main event! I’d been to the conference center on Monday to pick up my badge, but the volume of people that day didn’t even compare to how busy things got on Wednesday. It was a whirlwind of a day, and I spent some of my time at the Lightstep booth. The Lightstep crew did demos of the OTel Demo App to illustrate Observability-Landscape-as-Code in action, which Ana and I poured our blood, terror, terror, sweat, and (happy?) tears into it before KubeCon. It’s a great little demo, if I do say so myself, and I definitely recommend that you check it out. (Shameless plug, I know. Sorrynotsorry.)

The life-sized game of Jenga at our booth was a huge draw, and there were a lot of adventurous folks who tried their hand at it. The pic of Ana and me below shows how tall the tower got. What you don’t see is that, shortly after the pic was taken, one of us knocked the tower down. Oops. 😳

Our team also had KubeCon-Detroit-themed stickers made especially for this event. Ana took things to the next level and did her nails to match our stickers. And I don’t know if you can see it in any of the photos below, but she also wore Kubernetes-themed earrings too. She’s very extra, and I love it! The stickers were part of a scavenger hunt that revealed the location of a pizza party hosted by Lightstep on Thursday night.

Ana and I also spent some time roaming the exhibitors’ area, grabbing some sweet swag (I nabbed a sweet OTel t-shirt and the cutest little Tracetest plushie, whom I named Tracey), meeting new and interesting people, and meeting some familiar faces in real life. (Hi Abby!!) Ah, the life of a DevRel. So. Much. Fun. 💜💜💜

Unfortunately, I wasn’t able to stay for the full conference, but from what I did experience, it was lots of fun. This most definitely will not be my last KubeCon, and I most definitely can’t wait for the next KubeCon! Bring it!

Ana’s Point-of-View

Ana here! I wanted to provide my perspective from attending four KubeCons in the past. This is my second KubeCon through COVID, and I appreciate that the CNCF has enforced COVID protocols for their events. As someone who caught COVID during KubeCon EU València, I was extra careful this time around, and tested daily. I’m still testing and COVID negative, so thanks, CNCF!

On Monday, I was scheduled to go to the Contributor Summit, but I lost my voice the week before on Thursday, and I still hadn’t recovered it by Monday morning.🥲 I had talks and booth duty during the coming week, so instead I did something more easygoing and I just helped my fellow Lightsteppers get ready for KubeCon and did final touches on my slides.

One thing that was really hard this year was that there were too many (or what felt to me like too many) co-located/Day 0 events. It’s great to see the ecosystem continuing to grow, but it definitely makes it very hard to decide which events to go to, for an opportunity to learn from the community. I know I struggled with this, so I do hope that we try to unify topics a bit more next time.

On Tuesday, I split my time between Chaos Day, Keptn Community Day, and of course, the hallway track of the co-located events. From working in the Chaos Engineering community the last 4+ years, I loved to see the Cloud Native Chaos Engineering community growth, I also loved talking to folks about the future of Keptn, and giving a talk on how to achieve more reliability with this CNCF sandbox project and the other OSS tools.

As Wednesday kicked off, it was great to feel the buzz and excitement of folks so eager to learn and connect. From Day 1’s keynote, I really enjoyed the reminder that companies benefiting from Kubernetes and the CNCF ecosystem should be getting involved, giving back, and mentoring others. There was a lot of love to maintainers and contributors in the ecosystem, and as someone involved in the Kubernetes Release Team for Kubernetes v1.25 and v1.26, it was a huge honor to see my face up in there with the rest of the team! 💙

I also took some time to walk the show floor, buy some CNCF swag and check out the vendor swag too.

Thursday was another busy day of running around with the community. I also got to record some Cloud Native in Spanish content with some other Latinxs in the space. In addition to that, Lightstep hosted two events, one of which was our secret pizza party. It was fun to take over Detroit’s famous alley, The Belt, to hang out and talk about SRE. Thanks to everyone who stopped by! #DetroitSRECity 🤘

One of the other places I spent a lot of time at at KubeCon NA was the CNCF Project Pavillion. I was very happy to see that it was a bit larger than the area we had during KubeCon EU, but I still wish that it was bigger and wasn’t so tucked away in the corner. A number of booths were showcasing their projects with demos through the week, hosted Q&A time, and gave away swag. If you are still trying to understand the Cloud Native Ecosystem, you can look at this very extensive map of the landscape and projects under the CNCF, some of which are more advanced than others. Of course I’m biased, but I’m really excited for the work that Keptn is doing in helping developers have more control over their application lifecycle. I’m also very excited to see where Backstage goes and how other CNCF projects can integrate with their service catalog.

On Friday, I got a chance to hang out some more with my community friends and give a talk on the future of Keptn with our friends at Dynatrace. I am very excited to see where this project goes and what work we continue to do with OpenTelemetry to make it easier to observe our deployments and their reliability.

Final Thoughts

Aaaand...that’s a wrap! KubeCon was a ton of fun for both Ana and me. We loved having the Lightstep DevRel team in person for the first time, we loved connecting with our various CNCF communities, Lightstep customers, and it was great fun to see all the cool tech out there. Whether you’re a KubeCon newbie like me, or KubeCon veteran like Ana, it was great to see the CNCF community continuing to come together, grow, and prosper. We are definitely looking forward to KubeCon EU in 2023!

Now, please enjoy this photo of Adriana’s rat, Phoebe, keeping her company as she does her KubeCon expense report.

Peace, love, and code. 🦄 🌈 💫

Got questions about Observability? Talk to us! Feel free to connect with us through e-mail, or:

Connect with Adriana up on Twitter or LinkedIn
Connect with Ana on Twitter or LinkedIn.

Hope to hear from y’all!

Observability Mythbusters: Observability Anti-Patterns

Adriana Villela — Wed, 19 Oct 2022 19:57:23 +0000

Have you ever been at a place that claimed to be all in on Observability, only for you to realize that well, they’re not really following Observability practices? Yeah. Me neither. Just kidding. I’ve been in this space long enough to witness my fair share of anti-patterns, or what I like to refer to as, “Crimes Against Observability”. In today’s post, I’ll be calling these out, in the hopes that you can avoid these crimes and be on your merry way towards unlocking Observability’s powers.

Let’s get started!

1- Traces not treated as a first-class citizen

Too many organizations put far too much emphasis on metrics and logs, while either completely disregarding or downplaying traces. Yes, metrics and logs are useful…to a point.

Metrics can give us information about things like CPU levels and the amount of time that it takes to complete a transaction. But they can only provide aggregate information that you can’t drill down into to understand what’s going on with your system.

Logs provide useful point-in-time information; however, by themselves, logs make it pretty damn difficult to troubleshoot. They’re a wall of text that you have to parse through so you can kinda sorta maybe piece together what’s up with your code.

Neither metrics nor logs give you enough context to understand what’s happening with your system at a high level. Thus, the biggest crime against Observability is committed when metrics and logs are treated as the main actors of your Observability story, when in fact they take more of a supporting role. Spoiler alert: traces are the true stars of the show.

So how do we fix this? Take a trace-first approach. Traces give you that end-to-end system wide view. They show you not only what’s going on within services, but also across services. How do logs and metrics fit into this?

Make logs more useful by making them a part of our overall story—i.e. traces—embedded as Span Events.
Correlate metrics to traces via a linking attribute. For example, a VM with a given IP address can be correlated to a Trace if we capture IP address as a Span attribute.

Moral of the story: take a trace-first approach with your Observability landscape.

2- The Wall ’o Dashboards

Y’all, this one’s nails on chalkboard for me. I once worked at a place where leadership thought that their production woes would be solved by dashboarding all the things. “I helped set up a wall of dashboards when I worked at XYZ company and it helped us so much!” Yeah. Like 10 years ago, when there wasn’t much else to work with. Time to get with the times and rethink that wall ‘o dashboards, my friends.

Does that mean that dashboards go away entirely? No. Instead, rethink your dashboard situation. Use fewer dashboards. Don’t rely on the Wall ‘o Dashboards to guide your Observability journey. A better alternative to Metrics dashboards would be to use Service-Level Objectives (SLOs). SLOs are actionable. For example, suppose you have an SLO that states that the response time for Service X must be 95% of the time. If the service is not meeting that SLO, it triggers an alert to notify by Slack, phone, pager, passenger pigeon, or whatever, to tell your on-call engineers that your system is not behaving within the expected parameters, and that you've gotta take a closer look at things.

For more hot takes on dashboards, check out this great piece by Charity Majors and this short, fun video by Austin Parker.

3- Getting someone else to instrument your code

Say you’re a developer. Would you get someone else to comment your code or write your unit tests? I didn’t think so. Now, say that your team is getting into Observability. This means that you need to instrument your code à la OpenTelemetry. Would you:

Instrument your own code
Ask someone else (maybe your SREs?) to instrument your code

If you answered B, then, yeaaaaahhhh...“Houston, we have a problem”.

What’s wrong with this picture? Well, for starters, the SREs didn’t write the code. You did. Just like it would be super weird to have someone comment YOUR code and write unit tests for YOUR code, then why would it be okay to have someone else instrument YOUR code? How in Space do you expect them to know WHAT to instrument?

Look, there’s no shame in not having instrumented your code before. If you’re just getting started with Observability on an existing code base, you bet your pants that you’ll have to go through your code and instrument it. But you can also ensure that you instrument new code as you write it, going forward. Moral of the story, instrument your own code. More specifically, if you focus on instrumenting your home-grown frameworks and libraries, then you have all the coverage you need, as far as tracing is concerned. Whatever you do, please, don't get someone else to do your dirty work for you.

But hey, don’t take just my word for it. I’ll let Liz Fong-Jones have the last word:

“You’re a full grown-up software engineer. Write your own damn tests. Write your own damn comments. Write your own damn Observability annotations. This will help YOU understand your code later."

4- Belief that Observability Tooling == Observability

Oh, mes amis, this couldn’t be further from the truth. Observability is a set of practices supported by tools. These include:

Instrumenting your code properly so that you have enough info to troubleshoot when issues arise, and thereby avoiding calling the same group of “experts” to troubleshoot
Treating traces as first-class citizens
Keeping an eye on the health of your system after you release to prod
Creating meaningful, SLO-based alerts

Put that into place, and you’ve got yourself some Observability.

5- Observability theatre

We’ve seen it before. Companies going all-in on so-called “digital transformations”. Which, of course, includes going Agile and creating a DevOps practice. All noble causes. And the intentions are good…maybe? In reality, nobody actually believed it would work, or wanted it to work. In short, it was all theatre. And I’ve seen that with Observability too. “Hey! We need Observability!” Woo hoo! 🎉 But then it turns out to be a front for Land o’ Logs and Walls ‘o Metrics Dashboards. Because that’s what folks grew up with. And that’s all they want to know.

Moral of the story: if you don’t have buy-in from your org, both from the exec ranks and the engineers doing the work. Can you get said buy-in? Absolutely! It won’t be an easy journey, but the worthwhile things are never easy to achieve, are they?

Final Thoughts

Observability ain’t easy, but keeping the following things in mind will help guide you in your Observability journey:

Make Traces your first-class citizen in your Observability Landscape. ✅
Say “byeeeee” to the wall ’o dashboards, and hello to SLOs. ✅
Instrument your own damn code. Future You will thank you for it. ✅
Observability is a set of practices supported by tools. Sort your practices out before reaching out for the shiny new thing. ✅
Just say no to Observability Theatre.

And now, I shall reward you with a picture of an Octopus decked out for fall, drawn by my superly-talented 14-year-old daughter.

Peace, love, and code. 🦄 🌈 💫

Got questions about Observability and/or OpenTelemetry? Talk to me! Feel free to connect through e-mail, or hit me up on Twitter or LinkedIn. Hope to hear from y’all!

OpenTelemetry for Python: Manual Configuration & Context Propagation

Adriana Villela — Tue, 20 Sep 2022 20:11:24 +0000

In my last blog post, I showed y’all how to instrument Python code with OpenTelemetry (OTel), à la auto-instrumentation. You may also recall from that post that I recommended using the Python auto-instrumentation binary even for non-auto-instrumented libraries, because it abstracts all that pesky OTel config stuff so nicely. When you use it, along with any applicable Python auto-instrumentation libraries (installed courtesy of opentelemetry-bootstrap), it takes care of context propagation across related services for you.

All in all, it makes life nice ‘n easy for us!

Well, today, my friends, we’re going to torture ourselves a weeeee bit, because we’re going to put that auto-instrumentation binary aside, and will instead dig into super-duper manual OpenTelemetry instrumentation for Python. Since we don’t have auto-instrumentation as our security blanket, we will have to learn how to do the following:

Configure OpenTelemetry for Python to send instrumentation data to an Observability back-end that supports OTLP. Spoiler alert: we’ll be using Lightstep as our Observability back-end. ✅
Propagate context across related services so that they show up as part of the same trace ✅

Note: I won’t go into how to create Spans with OTel for Python, since the official OTel docs do a mighty fine job of it.

Are you scared? Well don’t be, because I’ve figured it all out so that you don’t have to!

Are you readyyyyy? Let’s do this!!

Pre-Requisites

Before we start our tutorial, here are some things that you’ll need:

A basic understanding of Python and Python virtual environments
A basic understanding of OpenTelemetry. I suggest checking out the official OTel Docs for refresher, if you need one.

If you’d like to run the full code examples in Part 2, you’ll also need:

A Lightstep Observability account
A Lightstep Access Token to tell Lightstep what project to send your traces to
A basic understanding of how to use Lightstep Observability
A working installation of Python

Part 1: What’s Happening?

We’ll be illustrating Python manual instrumentation with OpenTelemetry with a client and server app. The client will call a /ping endpoint hosted by the server.

The example in this tutorial can be found in the lightstep/opentelemetry-examples repo. We will be working with three main files:

common.py - OTel configuration and connectivity (to connect to Lightstep)
client.py - Connect to our server’s /ping endpoint
server.py - Host the /ping endpoint

Before we run the example code, we must first understand what it’s doing.

1- OTel Libraries

In order to send OpenTelemetry data to an Observability back-end (e.g Lightstep), you need to install the following OpenTelemetry packages, which are included in requirements.txt:

opentelemetry-api
opentelemetry-sdk
opentelemetry-exporter-otlp-proto-grpc

As you can see, we’re installing the OpenTelemetry API and SDK packages, along with opentelemetry-exporter-otlp-proto-grpc, which is used to send OTel data to your Observability back-end (e.g. Lightstep) via gRPC.

2- OTel Setup and Configuration (common.py)

In our example, OTel setup and configuration is done in common.py. We split things out into this separate file so that we don’t have to duplicate this code in client.py and server.py.

First, we must import the required OTel packages:

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

Next, we must configure the Exporter. An Exporter is how we send data to OpenTelemetry. As I mentioned earlier, Lightstep accepts data in the OTLP format, so we need to define an OTLP Exporter.

Note: Some vendors don’t accept data in OTLP format, which means that you will need to use a vendor-specific exporter to send data to them.

We configure our Exporter in Python like this:

def get_otlp_exporter():
   ls_access_token = os.environ.get("LS_ACCESS_TOKEN")
   return OTLPSpanExporter(
       endpoint="ingest.lightstep.com:443",
       headers=(("lightstep-access-token", ls_access_token),),
   )

Some noteworthy items:

The endpoint is set to ingest.lightstep.com:443, which points to Lightstep’s public Microsatellite pool. If you are using an on-premise satellite pool, then check out these docs.
You will need to set the LS_ACCESS_TOKEN environment variable with your own Lightstep Access Token.

Finally, we configure the Tracer Provider. A TracerProvider serves as the entry point of the OpenTelemetry API. It provides access to Tracers. A Tracer is responsible for creating a Span to trace the given operation.

We configure our Tracer Provider in Python like this:

def get_tracer():
    span_exporter = get_otlp_exporter()

    provider = TracerProvider()
    if not os.environ.get("OTEL_RESOURCE_ATTRIBUTES"):        
        # Service name is required for most backends
        resource = Resource(attributes={
            SERVICE_NAME: "test-py-manual-otlp"
        })
        provider = TracerProvider(resource=resource)
        print("Using default service name")

    processor = BatchSpanProcessor(span_exporter)
    provider.add_span_processor(processor)
    trace.set_tracer_provider(provider)    

    return trace.get_tracer(__name__)

A few noteworthy items:

We define a Resource to provide OpenTelemetry with a bunch of information that identifies our service, including service name and service version. (You can see a full list of Resource attributes that you can set here.) As the name implies, service name is the name of the microservice that you are instrumenting, and service version is the version of the service that you are instrumenting. In this example, we get the service name and service version are passed in as key/value in the environment variable, OTEL_RESOURCE_ATTRIBUTES (we’ll see some example values in Part 2). If that environment variable is not present, we then set a default service name, "test-py-manual-otlp".
We are using the BatchSpanProcessor, which means that we are telling OTel to export the data in batches. For the purposes of this example, we’re not doing anything beyond a basic configuration.

3- Initialization (client.py and server.py)

We’re finally ready to send data to Lightstep! All we need to do is call common.py’s get_tracer function from client.py (Lines 17-20) and server.py (Lines 17 and 29), like this:

from common import get_tracer

...

tracer = get_tracer()

...

4- Instrumentation (client.py and server.py)

With initialization done, we need to instrument our code, which means that we’ll need to create Spans. I won’t go into the specifics of Span creation here, since the OTel docs do a pretty good job of it, and as I mentioned in the intro, it’s outside of the scope of this post.

I will, however, briefly mention that there are a couple of ways to instrument our code in Python, and you’ll see both ways of Span creation in the example code: using the with statement, and using function decorators.

You can see an example of creating a Span using the with statement in client.py, Lines 23-32. Below is the full function listing:

def send_requests(url):
    with tracer.start_as_current_span("client operation"):
        try:
            carrier = {}
            TraceContextTextMapPropagator().inject(carrier)
            header = {"traceparent": carrier["traceparent"]}
            res = requests.get(url, headers=header)
            print(f"Request to {url}, got {len(res.content)} bytes")
        except Exception as e:
            print(f"Request to {url} failed {e}")
            pass

The Span is initialized with the line, with tracer.start_as_current_span("client operation"):, and everything below that line is within the scope of that Span.

You can see an example of creating a Span using a function decorator in server.py Line 78. Below is the full function listing:

@tracer.start_as_current_span("pymongo_integration")
@app.route("/pymongo/<length>")
def pymongo_integration(length):
    with tracer.start_as_current_span("server pymongo operation"):
        client = MongoClient("mongo", 27017, serverSelectionTimeoutMS=2000)
        db = client["opentelemetry-tests"]
        collection = db["tests"]
        collection.find_one()
        return _random_string(length)

A few noteworthy items:

The line @tracer.start_as_current_span("pymongo_integration") starts the Span for the pymongo_integration function. Everything in that function is within the scope of that Span.
You may have also noticed that we initialize another span in there, with the line, with tracer.start_as_current_span("server pymongo operation"):, (server.py, Line 89). This means that we end up with nested Spans (a Span within a Span).

5- Context Propagation

As I mentioned in the intro, one of the advantages of using Python auto-instrumentation is that it takes care of context propagation across services for you. If you don’t use auto-instrumentation, however, you have to take care of context propagation yourself. Great. Just great.

But before we dig into how to do that, we need to first understand context propagation.

Definition time!

Context represents the information that correlates Spans across process boundaries.

Propagation is the means by which context is bundled and transferred in and across services, often via HTTP headers.

This means that when one service calls another, they will be linked together as part of the same Trace. If you go the pure manual instrumentation route (like we’re doing today), however, you have to make sure that your context is propagated across services that call each other, otherwise you’ll end up with separate, unrelated-even-though-they-should-be-related) Traces.

I have to admit that I was wracking my brains trying to figure out this context propagation stuff. After much time spent Googling and asking folks around here for clarification, I finally got it, so I’m going to share this piece with you here to hopefully spare you some stress.

Note: Although the OpenTelemetry documentation does provide some insight into how to do manual context propagation in Python, the documentation needs a little work. I’m actually part of the OpenTelemetry Comms SIG, so I am using this as motivation to improve the docs around this topic…stay tuned for updates to the OTel docs too! 😎

Okay, so how do we do this manual context propagation? First, let’s remind ourselves of what’s happening in our example app. We have a client service and a server service. The client service calls the /ping endpoint on the server service, which means that we expect them to be part of the same Trace. This in turn means that we have to ensure that they both have the same Trace ID in order to be seen by Lightstep (and other Observability back-ends) as being related.

At a high level, we accomplish this by:

Getting the Trace ID of the client
Injecting the Trace ID into the HTTP header before the client calls the server
Extracting the client’s Trace ID from the HTTP header on the server side

Easy peasey! Now let’s look at the code that needs to make this happen.

First, we need to start with something called a carrier. A carrier is just a key-value pair containing a Trace ID, and it looks something like this:

{'traceparent': '00-a9c3b99a95cc045e573e163c3ac80a77-d99d251a8caecd06-01'}

Where traceparent is the key, and the value is your Trace ID. Note that the above is just an example of what a Trace ID might look like. Obviously, your own Trace ID will be different (and will be different each time you run the code).

Okay, great. Now how do we obtain said carrier?

First, we need to import a TraceContextTextMapPropagator in client.py:

from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

Next, we must populate the carrier:

carrier = {}
TraceContextTextMapPropagator().inject(carrier)

If you were to inspect the value of carrier after this line, you would see that it would look something like this:

{'traceparent': '00-a9c3b99a95cc045e573e163c3ac80a77-d99d251a8caecd06-01'}

Look familiar? 🤯

Now that we have the carrier, we need to put it into our HTTP header before we make a call to the server.

header = {"traceparent": carrier["traceparent"]}
res = requests.get(url, headers=header)

And voilà! Your carrier is in the HTTP request!

Now that we know what all of these snippets do, let’s put it all together. Here’s what our client code looks like:

def send_requests(url):
    with tracer.start_as_current_span("client operation"):
        try:
            carrier = {}
            TraceContextTextMapPropagator().inject(carrier)
            header = {"traceparent": carrier["traceparent"]}
            res = requests.get(url, headers=header)
            print(f"Request to {url}, got {len(res.content)} bytes")
        except Exception as e:
            print(f"Request to {url} failed {e}")
            pass

For the full code listing, check out client.py.

Okay…we’ve got things sorted out on the client side. Yay! Now let’s go to the server side and pluck our carrier from the HTTP request.

In server.py, we pull the value of traceparent from our header like this:

traceparent = get_header_from_flask_request(request, "traceparent")

Where we define get_header_from_flask_request as:

def get_header_from_flask_request(request, key):
    return request.headers.get_all(key)

Now we can build our carrier from this information:

carrier = {"traceparent": traceparent[0]}

We use that to extract the context from this carrier:

ctx = TraceContextTextMapPropagator().extract(carrier)

Now we can create our Span with the context, ctx:

with tracer.start_as_current_span("/ping", context=ctx):

Here, we are passing ctx to a named parameter called context. This ensures that our "/ping" Span knows that it’s part of an existing Trace (the one originating from our client call).

It is worth noting that any child Spans of the "/ping" Span do not require us to pass in a context, since that’s passed in implicitly (see server.py, Line 81, for example).

Now that we know what all of these snippets do, let’s put it all together. Here’s what our server code looks like:

...

from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

...

def get_header_from_flask_request(request, key):
   return request.headers.get_all(key)

...

@app.route("/ping")
def ping():

   traceparent = get_header_from_flask_request(request, "traceparent")
   carrier = {"traceparent": traceparent[0]}   
   ctx = TraceContextTextMapPropagator().extract(carrier)

   with tracer.start_as_current_span("/ping", context=ctx):

       length = random.randint(1, 1024)
       redis_integration(length)
       pymongo_integration(length)
       sqlalchemy_integration(length)
       return _random_string(length)

...

For the full code listing, check out server.py.

Part 2: Try it!

Now that we know the theory behind all of this, let’s run our example!

1- Clone the repo

git clone https://github.com/lightstep/opentelemetry-examples.git

2- Setup

Let’s first start by setting up our Python virtual environment:

cd python/opentelemetry/manual_instrumentation

python3 -m venv .
source ./bin/activate

# Install requirements.txt
pip install -r requirements.txt

3- Run the Server app

We’re ready to run the server. Be sure to replace <LS_ACCESS_TOKEN> with your own Lightstep Access Token.

export LS_ACCESS_TOKEN="<LS_ACCESS_TOKEN>"
export OTEL_RESOURCE_ATTRIBUTES=service.name=py-opentelemetry-manual-otlp-server,service.version=10.10.9

python server.py

Remember how I told you that we’d see an example of values passed into OTEL_RESOURCE_ATTRIBUTES? Well, here it is! Here, we’re passing in the service name py-opentelemetry-manual-otlp-server, and service version 10.10.9. The service name will show up in the Lightstep explorer.

Your output will look something like this:

4- Run the Client app

Open up a new terminal window, and run the client app. Be sure to replace <LS_ACCESS_TOKEN> with your own Lightstep Access Token.

PS: Make sure you’re in python/opentelemetry/manual_instrumentation in the opentelemetry-examples repo root.

export LS_ACCESS_TOKEN="<LS_ACCESS_TOKEN>"
export OTEL_RESOURCE_ATTRIBUTES=service.name=py-opentelemetry-manual-otlp-client,service.version=10.10.10

python client.py test

Note how we’re passing in the service name py-opentelemetry-manual-otlp-client, and service version 10.10.10. The service name will show up in the Lightstep explorer.

When you run the client app, it will continuously call the /ping endpoint. Let it run a few times (maybe 5-6 times-ish?), and kill it (à la ctrl+c). Sample output:

If you peek over at the terminal running server.py, you will likely notice a super-ugly stack trace. DON’T PANIC! The /ping service makes calls to Redis and MongoDB, and since neither of these services is running, you end up getting some nasty error messages like this:

5- See it in Lightstep

If you go to your trace view in Lightstep by selecting the py-opentelemetry-manual-otlp-client service from the explorer (you could also see the same thing by going to the py-opentelemetry-manual-otlp-server service), you’ll see the end-to-end trace showing the client calling the server, and the other functions called within the server.

And remember that stack trace in Step 4? Well, it shows up as an error in your Trace. Which is cool, because it tells you that you have a problem, and pinpoints to where it’s happening! How cool is that??

And remember how we never passed our context to the redis_integration and server redis operation Spans, you can see that server redis operation rolls up to redis_integration, which rolls up to /ping, just like I said it would. Magic! 🪄

Final Thoughts

Today we learned how to manually configure OpenTelemetry for Python to connect to Lightstep (this also works for any Observability back-end that ingests the OTLP format). We also learned how to link related services together through manual context propagation.

Now, if you ever find yourself in a situation whereby you need to either connect to your Observability back-end without the use of the Python auto-instrumentation binary and/or need to manually propagate context across services, you will know how to do it!

Now, please enjoy this cuddly little pile of rats. From front to back: Phoebe, Bunny, and Mookie. They were nice enough to sit still for the camera while my husband held them.

Peace, love, and code. 🌈 🦄 💫

Got questions about OTel instrumentation with Python? Talk to me! Feel free to connect through e-mail, or hit me up on Twitter or LinkedIn. Hope to hear from y’all!

Introducing the On-Call Me Maybe Podcast!

Adriana Villela — Mon, 12 Sep 2022 13:18:43 +0000

Y’all, I am so excited to announce that Ana Margarita Medina and I just launched the On-Call Me Maybe podcast! This podcast is dedicated to nerding out on all things DevOps, SRE, Observability, On-Call, and everything in between. Want to learn more? Then read on, my friend!

Your most awesome hosts

Ana and I met when we both joined Lightstep as Developer Advocates in April of this year, and though we haven’t known each other for very long, we are very much partners in crime.

I’m super excited about this collaboration because Ana is a fellow Latina in tech. Being in a male-dominated industry, it’s been beyond super cool to work with a fellow tech gal from a similar cultural background. I was originally born in Rio de Janeiro, Brazil, and moved to Canada when I was 10. Ana hails from Costa Rica, and her parents are from Nicaragua. She moved to the US when she was in elementary school. We bring some of the flair of our respective backgrounds to this podcast. 😎

Our guests

We want to use this podcast as a platform for nerdy tech discussions, sociotechnical systems, mental health, and diversity & inclusion…and our guests reflect that! We want to connect our audience with folks from all walks of tech. You’ll hear from some familiar faces, and some not-so-familiar faces. Our conversations are fun and casual, with chill vibes all around.

Give us a listen!

Our first episode dropped on September 6th. It features a conversation between Ana and me, as we talk about our beginnings in tech. We’ll be coming at you with new episodes on a weekly basis. You’ll be hearing from folks at Honeycomb, Jeli, ThoughtWorks, HashiCorp, Wavelo, and more! We’ve also got a couple of guest hosts, including Alex Boten and Ted Young.

So what are you waiting for? Be sure to give us a listen, follow On-Call Me Maybe on Twitter, follow our LinkedIn page, and subscribe via your favourite podcasting app!

Peace, love, and code. 🌈 🦄 💫

Let's Talk About Psychological Safety

Adriana Villela — Thu, 01 Sep 2022 17:29:44 +0000

There’s something askew…

On the morning of Friday, July 8th, 2022, as I rolled out of bed, I instinctively reached over to my phone to check the weather for the day. Right away, I knew that something was off. My weather app wouldn’t load the weather. I toggled my wifi only to realize that I had no bars of cell signal on my phone. My husband’s phone also had no bars. No cell service and no internet. What was this? The ‘80s?!

As remote workers, this meant that we could not work until this thing got resolved. Cut off from the outside world, we scrambled to find out what was up. How bad was this thing? Luckily we live near a subway station with free wifi, so we stood across the street from the station and bummed a wifi signal from there.

We were able to check in with our respective co-workers and scour the interwebs to find out what was up. There was a Rogers service outage, and it affected cable, cell, and internet services. For those not in Canada, Rogers is one of our three large telecom providers. Our other two large providers, Telus and Bell, were unaffected.

Psychological Safety

To say that this situation sucked is an understatement. It definitely made me realize just how dependent we are on internet connectivity for our day-to-day lives. Blows my mind. 🤯 I wasn’t able to work at all on Friday, which meant I was a bit behind on my work. I know that it was far worse for others. I think of the Uber drivers who couldn’t make their living because they were on the affected network. I think of the store owners who couldn’t accept debit payments because the outage affected Interac. What about folks closing on a house that day who couldn’t transfer funds? Folks who needed to cash out investments to make a down payment for a house. Or folks who couldn’t get cash at the ATMs of the affected banks. So yeah. This definitely sucked for a LOT of folks. But do you know who this also sucked for? The folks in the thick of this outage, who were probably breaking out in cold sweats, shaking in their booties, losing sleep, skipping meals, and straight up having panic attacks as they tried to resolve this issue as quickly as possible.

Before y’all come at me with your torches and pitchforks, let me remind you that the folks who were working to resolve Friday’s outage are HUMAN, and they were probably having the WORST. DAY. EVER.

This got me thinking of all the times I was involved with major production issues, and just how friggin’ STRESSFUL it was. Imagine being called on to solve a production issue. You don’t know what’s going on. You need to get your bearings, and you need to do it FAST. To top it all off, you’re likely on a call with a ton of people, including project managers and execs who are breathing down your neck, asking you for updates. If you’re super lucky, you’ll get some cowboy exec who “used to code” 20 years ago and tries to tell you how to troubleshoot. Recipe for SO. MUCH. FUN. Not.

Many folks who have worked in support or operations type roles have some serious PTSD from their experiences when they are called into fixing production issues. To the point where some just walk away from these types of roles, because it scars them for life. And the reason why they have PTSD over this is because they don’t have psychological safety.

So, what is psychological safety? Wikipedia actually gives a really good definition:

Meh. Who cares?

Why is psychological safety important? Because it gives folks room to fail. I’ve put in changes that broke a system before. It wasn’t intentional. And when I realized what happened, I was wracked with guilt, and was sweating buckets to find out the root cause. Unfortunately, like many folks in our industry, I didn’t have sympathetic management on my side to shield me from the fallout. If anything, I’ve had managers who were only too happy to point fingers at me for my failures.

Unfortunately, even though I vowed to not do this myself as a manager, I’m ashamed to say that I’ve been part of the problem too. I was once managing a team that was responsible for a critical backbone system. They made a maintenance change during the day, assuming that it wouldn’t affect anything, because when they’d made the same change in pre-prod, everything had been copacetic. Of course, things went south in prod. Because you know…Murphy. It brought the system down. Fortunately, my team was amazing and they were able to figure out the issue quickly, and brought things up within minutes. High fives all around.

Then, upper management started messaging me. And texting me. They wanted answers. Why did this system go down? How could this happen? I assured them that the team didn’t realize that this would happen in prod, since it had worked in pre-prod. I promised that we would use off-hours maintenance windows. The team was not happy. We were trying to push the needle by following more SRE-like practices (like deployments and updates during business hours), and this was not SRE-like at all. We all knew it. But I was trying to keep upper management happy.

We had another outage a couple of weeks later. Similar, but different. Fixed quickly again. But upper management was livid. Why was this happening again? More messages, more texts, and video calls to the tone of “Explain yourself!”

I let them get into my head, and I told the team how upper management was unhappy with these outages. Which of course stressed my team. They felt bad enough about the outage as it was. They’re good people. Smart people. Trying to do the right thing. But they’re also human. They make mistakes. And getting flack from me, even if it was minor flack, was distressing. And my team lead called me out on it. He sent me a private message saying that I wasn’t providing the team with psychological safety.

Whoa. Did that ever hit me like a ton of bricks! My first reaction was anger. How rude! How dare he? I thought I went pretty easy on the team. My tone was chill. What’s his beef? They weren’t following protocol. Screw him. I was livid.

But now that I think of it, HE. WAS. RIGHT.

You see, there were two points of failure here. First, upper management was putting pressure on me because their major client had experienced so many outages (mostly not related to our team) that just one more outage, even a minor one that was fixed quickly, eroded their confidence in the company. But I think that perhaps they could’ve “trained” their client to understand that outages happen and that it’s about how quickly you recover from the outages that matters. Instead, they continued to feed the client’s notion that failure should never ever ever happen. If only…

Then there was me. As a manager and spokesperson for my team, I should’ve done a better job of protecting them. Making it safe for them to fail. Telling them things like, “Upper management is on my butt over this and they want answers” just makes a stressful situation worse. I should’ve also pushed harder on the notion that we can only mature our SRE practices if we’re allowed to fail fast, recover quickly, and make changes during business hours. As a leader, I have to help make the change happen, and in this case I didn’t.

Do I regret my actions? Definitely. Couldda wouldda shouldda, right? But let me tell you...I learned from my gaffe. I am so much more aware of psychological safety as both a manager and an individual contributor, and as part of my mission as a Developer Advocate, I want more people talking about psychological safety.

The Path to Psychological Safety

How do we achieve psychological safety? Though behavioural changes. That good ‘ole mindset change that we hear in the DevOps and Observability circles! Yes, it’s a real thing! And it starts with leadership.

For leaders, here are some things y’all can do to help:

Give engineers room to breathe. I can assure you that asking the poor engineer when the issue will be fixed is incredibly aggravating. I can also assure you that when they have an update, the engineer will happily, gleefully, and excitedly be more than willing to report status.
Don’t make your teams feel guilty for causing a failure or not fixing things quickly enough. Remember that sometimes you have people who had nothing to do with the failure who are trying really to fix that system. Y’all need to cut them some slack!
Understand the root cause. This means that you don’t point fingers at who caused the change, but instead focus on the what – i.e. understand the contributing factors that led to the failure.
Create a safe space for your direct reports. In doing so, they will feel comfortable in coming to you with concerns and even suggestions for improvement. To achieve this safe space, you will need to invest in relationship-building. Trust in them, and in return, they will trust you. If they feel that you have their back, they’ll have yours. I assure you. I’ve been on both sides.
Celebrate your engineers for the rockstars that they are. (Maybe give ‘em a hug too. #HugOps) Really. As one friend of mine put it, 99% of the time, we as end-users are spoiled by relatively smooth-running systems, because there are engineers working their magic to minimize service disruptions. And yet folks are ready to pick on the 1% that went wrong, without even appreciating that there’s some seriously amazing work being done 99% of the time. How’s that good for morale, retention, and burnout prevention when your staff don’t even feel appreciated?
**Embrace the failure. **Failure is inevitable, so the sooner you learn to deal with it, the better!

So the above list is a great start. But wait…there’s more that can be done! Because we have some wonderful tools and practices at our disposal to help make this happen.

First off, there’s Observability. (Come on…y’all didn’t think I could do a whole blog post without mentioning Observability, did you? 😉) Observability is your friend when major incidents arise, because it provides you a holistic view of your system. Properly-instrumented code, along with a good Observability back-end help you see what’s going on in your systems so that you can troubleshoot things quickly and effectively. In doing so, troubleshooting goes from, “Ahhhh!! WTF is happening?!” to, “Oh, I can follow these breadcrumbs to see what’s going on in my system.” This puts support folks at ease because they know that they have the tools to confidently troubleshoot an issue. This puts management at ease, because they know that their support folks are empowered to troubleshoot quickly and effectively. In short, Observability gives you psychological safety.

// Detect dark theme var iframe = document.getElementById('tweet-1545162437132820480-187'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1545162437132820480&theme=dark" }

// Detect dark theme var iframe = document.getElementById('tweet-1545163665661362178-551'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1545163665661362178&theme=dark" }

Secondly, there are Service-Level Objectives (SLOs). SLOs help us identify what data ingested by your Observability back-end are important and alert-worthy. SLOs connect telemetry with specific customer experiences, so that the entire organization can understand the relationship between the software system and the business goals.

Finally, there is Incident Response tooling, which, when properly configured and integrated with your Observability back-end, ensures that the right people are contacted at the right time, triggered by the right SLO-based alerts to investigate an issue.

In a nutshell, psychological safety is made possible by:

Proper support from management
Observability practices supported by tooling
SLO-based alerts
Incident response processes and tooling

Final Thoughts

I think we can all agree that the July 8th/9th outage sucked for those of us affected. This also exposes the fact that there’s a system resiliency and reliability issue that needs to be addressed in order to ensure customer retention and confidence. I don’t think that anyone would argue with that. But remember that our systems of today are far more complex than they were even 2 years ago. The systems and devices that we enjoy today are backed by impossibly complex systems, and the more complex our systems, the more risk is involved.

Also, remember that this situation also TOTALLY SUCKED for the folks on the front lines who were desperately trying to resolve the Rogers outage. That said, if anything good came out of it, it’s that it once again opens up the conversation about psychological safety and how important it is in the tech world. Without psychological safety, you end up with burnout, and you end up with fewer and fewer people who are willing to take on these critical support roles that make our world run seamlessly.

As a final thought, I’d like to leave you with these words from a good friend of mine:

Now, please enjoy this picture of my rat Chrissy, who fancies herself a lioness today.

Peace, love, and code. 🦄 🌈 💫

Join the conversation! We’d love to hear your thoughts on psychological safety. Connect with us on the Lightstep Community Discord, or get in touch by e-mail. Hope to hear from y’all!

Observability Mythbusters: How hard is it to get started with OpenTelemetry?

Adriana Villela — Mon, 29 Aug 2022 15:31:00 +0000

So you're new to Observability and OpenTelemetry and Lightstep and you want to send traces to Lightstep to get a feel for the flow of things. But you're not sure how to get started. There's a LOT to learn and take in, and things can get pretty overwhelming, pretty fast, amirite? I feel ya. I've been there before. That was me last year. Solidarity!!

But, not to worry, my friend, because I am here to walk you through things. If you're looking to do a quickie setup to learn how to send OpenTelemetry data over to Lightstep, then you, my friend, have come to the right place!

The OpenTelemetry Demo repo was created to help lower the barrier to entry into OpenTelemetry and to illustrate its awesomeness. It features a fully-functioning online store made up of multiple micro-services, written in different languages that are supported by OpenTelemetry, that can send traces to an Observability back end, like Lightstep. (OMG that was a mouthful!) Most importantly, I found it superly duperly easy to set up and get going!

In today's tutorial, I will show you how to get Traces produced by the OpenTelemetry Demo into Lightstep. The example uses the OpenTelemetry (OTel) Collector to ingest the Traces and send them to an Observability back-end, which in our case, will be Lightstep.

A few important notes:

I am assuming that you have a basic understanding of OpenTelemetry and Observability. If not, check out this great OpenTelemetry overview and Observability oberview.
I will not be covering application instrumentation with OpenTelemetry - our goal is merely to get Trace data into Lightstep via the OTel Collactor.

Before we start the tutorial, let's do a quick OTel Collector overview, so that you can understand how you're getting data into Lightstep!

OTel Collector 101

The OpenTelemetry (OTel) Collector is a vendor-neutral service that ingests, transforms, and exports data to one or more Observability back-ends.

The OTel Collector consists of 3 main components:

Receivers ingest data. Example data sources include: Kubernetes clusters, Prometheus, and application code.
Processors transform data. This can include adding/removing data, modifying data (e.g. data masking), and even filtering data.
Exporters send data to an Observability back-end. In our case, Lightstep. Lightstep ingests OTel data natively, using the OpenTelemetry Protocol (OTLP).

Pipelines are used to define how data flows in and out of the OTel Collector.

For more on the OTel Collector, please see the official OTel docs site.

NOTE: Although your instrumented code could technically send data directly to an Observability back-end (e.g. Lightstep) without a Collector, it's not really the right way to go about it. You should always funnel all OTel data (regardless of the soure) to your back-end by way of the OTel Collector.

Running the OTel Collector

There are several ways to deploy the OTel Collector; however, we're keeping things simple and local today, using Docker Compose, which runs Docker.

Configuration

The OTel Collector is configured via YAML.

The YAML file is used to define receivers, processors, exporters, and pipelines.

Tutorial

Ah, now the moment y'all have been waiting for - the tutorial! Let's do this!

For this tutorial, we will be using a modified version of the OpenTelemetry Demo repo. This repo has been modified to demonstrate how to send data to Lightstep using the OTel Collector. While the upstream repo contains configs to send data to a local installation of Jaeger, the modified version contains configs to send data to Lightstep. We will be keeping the Lightstep repo in sync with the upstream repo.

Pre-Requisites

Running the Demo App Locally

1- Clone the repo

git clone https://github.com/lightstep/opentelemetry-demo.git

2- Edit the OTel Collector Config file

This file allows you to configure the OpenTelemetry Collector with your own Observability back-end. This case, otelcol-config-extras.yml is already configured to use Lightstep as the back-end. All you need to do is add your Lightstep Access Token to the file, to be able to send traces to Lightstep.

cd opentelemetry-demo

Open src/otelcollector/otelcol-config-extras.yml for editing using your favourite editor. The file looks like this:

# extra settings to be merged into OpenTelemetry Collector configuration
# do not delete this file
exporters:
  logging:
    logLevel: debug
  otlp/ls:
    endpoint: "ingest.lightstep.com:443"
    headers:
      "lightstep-access-token": <lightstep_access_token>

service:
   pipelines:
     traces:
       receivers: [otlp]
       processors: [batch]
       exporters: [logging, otlp/ls]
     metrics:
       receivers: [otlp]
       processors: [batch]
       exporters: [prometheus, logging, otlp/ls]

Replace <lightstep_access_token> with your own Lightstep Access Token, and save the file. The Access Token tells what Lightstep project to send your telemetry data to.

A few noteworthy items:

Lightstep ingests data in native OpenTelemetry Protocol (OTLP) format, so we will use the OTLP Exporter. The exporter can be called either otlp or follow the naming format otlp/<something>. We could call it otlp/bob if we wanted to. We're calling our exporter otlp/ls to signal to us that we are using the OTLP exporter to send the data to Lightstep.
Though not mandatory, we are also using a Logging Exporter. This is helpful, as it prints our traces to the Collector's stdout. As you might imagine, this is great for debugging, especially if you see traces sent to stdout, but not to your back-end.
We must define a pipeline in the service.pipelines section of the YAML config. Specifically, we need to define a pipeline for our taces. The pipeline tells the Collector:
- Where it's getting Trace data from (it's being sent via OTLP)
- If there's any processing that needs to be done (this is optional)
- Where to send data to. In our case, it's to stdout (via the Logging Exporter) and to Lightstep (via OTLP Exporter)
The Collector can ingest data using both HTTP and gRPC. The receivers configuration may appear to be empty; however, it actually means that we are using the defult values for the receivers config. It is actually the same as saying:

receivers:
   otlp:
     protocols:
       grpc:
         endpoint: 0.0.0.0:4317
       http:
         endpoint: 0.0.0.0:4318

TIP: If you find that you can't see data in Lightstep, make sure that you've defined a pipeline, and that the pipeline's exporter lists your otlp/ls exporter. I can tell you that twice this week I forgot to add my exporter to the pipeline, and was sitting in front of my computer in sheer panic trying to figure out why I could see my traces in stdout but not in Lightstep. 😱

3- Launch the sample app

If you're not already in the repo root, make sure you go there:

cd opentelemetry-demo

Run Docker compose. If you’re on an x86-64 machine (i.e. Intel or AMD-based, non-Apple Silicon), run the command below. If you’re running the Demo App for the first time, it will download all of the Docker images first. This may take several minutes.

docker compose up --no-build

If you’re on an Apple Silicon machine, you will need to build all images from source using the command below, as the images hosted in the GitHub Container Registry (GHCR) are built and optimized for x86-64 machines. This can take upwards of 20 minutes.

docker compose build
docker compose up

Once the app is up and running, you will see text continuously scrolling down the screen. This is expected!! The sample output looks something like this:

ASIDE: After the app started up, I kept waiting for the scrolling text to stop, only to realize that, "Oh, this thing is running. Guess I can access the URL now!" 🙄

Since we added our Logging Exporter to our pipeline definition, you'll also see that zoom by past you in the terminal:

NOTE: Obviously the Logging Exporter is more useful when it's not running with a bunch of other services competing for your attention on your terminal. This will be more useful if you're running it in Kubernetes or Nomad, or running standalone with Docker.

You can now access the Demo app: http://localhost:8080

Be sure to play around. Browse. Add items to your cart. Remove items from your cart. Checkout.

4- See the Traces in Lightstep

To view traces in your Lightstep Observability project, check out Lightstep Observability Notebooks.

And that's it! Didn't I tell you? Pretty straightforward!

Conclusion

Today we saw how easy it was to get Trace data into Lightstep using the modified version of the OpenTelemetry Demo Webstore, which uses our good friend, the OTel Collector.

Now that you've been able to get Trace data into Lightstep, I encourage you to play around with it. If you have any questions about Observability or OTel, pop into the Lightstep Community Discord, or get in touch by e-mail.

And to reward you for your work today, please enjoy this cartoon drawn by my daughter.

Peace, love, and code. 🦄 🌈 💫

The OpenTelemetry Demo Webstore App is always looking for feedback and contributors. Please consider joining the OTel Community to help make OpenTelemetry AWESOME!

Auto-Instrumentation is Magic: Using OpenTelemetry Python with Lightstep

Adriana Villela — Mon, 22 Aug 2022 15:04:00 +0000

In my last OpenTelemetry blog post, I talked about how to send OpenTelemetry (OTel) data to Lightstep using Golang. That’s all well and good if you’re a Golang developer, but what if you use Python? Well, my friend, you’re in luck, because today, I’ll be looking at how to send OpenTelemetry data to Lightstep using Python.

As with the OTel Golang post, we can send OTel data to Lightstep (or any other Observability tool that supports OpenTelemetry Protocol (OTLP), for that matter) in one of 3 ways:

Direct from application
OpenTelemetry Collector
Launchers

In this post, I will dig into each of these three approaches in detail, with code snippets which explain how to get data into Lightstep Observability. Let’s do this!

OpenTelemetry & Lightstep

Lightstep Observability supports the native OpenTelemetry Protocol (OTLP). It can receive data in the OTLP format either via HTTP or gRPC. You will need to specify which method you wish to use in your code, as we’ll see in the upcoming code snippets.

If you're curious about using gRPC vs HTTP for OpenTelemetry, check out these docs.

Note: Other Observability tools that support OTLP include Honeycomb, Grafana, and Jaeger.

Automatic Instrumentation & Python

One thing that’s super cool about using OTel to instrument your Python code is that Python offers automatic (auto) instrumentation. What does this mean? At a high level, it means that you can run a Python OpenTelemetry binary (called opentelemetry-instrument) that wraps around your Python application, to automagically instrument it. 🪄

More specifically, auto-instrumentation uses shims or bytecode instrumentation agents to intercept your code at runtime or at compile-time to add tracing and metrics instrumentation to the libraries and frameworks you depend on. The beauty of auto-instrumentation is that it requires a minimum amount of effort. Sit back, relax, and enjoy the show. A number of popular Python libraries are auto-instrumented, including Flask and Django. You can find the full list here.

Manual instrumentation requires adding spans, context propagation, attributes, etc. to your code. It’s akin to commenting your code or writing tests.

Does this mean that you shouldn’t manually instrument? Not at all! Start with auto-instrumentation if it’s available. If the auto-instrumentation isn’t sufficient for your use case (most often it’s not), then add in the manual instrumentation. For example, auto-instrumentation doesn’t know your business logic—it only knows about frameworks and languages—in which case you’ll want to manually instrument your business logic, so that you get that visibility.

Pre-Requisites

Before we start our tutorial, here are some things that you’ll need:

A basic understanding of Python and Python virtual environments
A basic understanding of the OpenTelemetry Collector
A basic understanding of how to use Lightstep Observability

If you’d like to run the full code examples, you’ll also need:

A Lightstep Observability account
A Lightstep Access Token to tell Lightstep what project to send your traces to
A working installation of Python
Docker (to run the OTel Collector locally)

Direct from Application

If you’re getting started with instrumenting your application with OpenTelemetry, this is probably the most common route taken by most beginners. As the name suggests, we are sending data to a given Observability back-end directly from our application code.

Our sample application is a Flask application. We will be leveraging both automatic and manual instrumentation.

Let’s look at this in greater detail below.

1- Set up your environment

Let’s set up our working directory and our Python virtual environment

mkdir otel_python
cd otel_python

python3 -m venv .
source ./bin/activate
touch server.py

Open server.py, and paste the following:

from flask import Flask, request
from opentelemetry import trace
from random import randint
tracer = trace.get_tracer_provider().get_tracer(__name__)

app = Flask(__name__)

@app.route("/rolldice")
def roll_dice():
    return str(do_roll())

@tracer.start_as_current_span("do_roll")
def do_roll():
    res = randint(1, 6)
    current_span = trace.get_current_span()
    current_span.set_attribute("roll.value", res)
    current_span.set_attribute("operation.name", "Saying hello!")
    current_span.set_attribute("operation.other-stuff", [1, 2, 3])
    return res

if __name__ == "__main__":
    app.run(port=8082, debug=True, use_reloader=False)

2- Install the required OTel libraries

These are the libraries that are required to send data to an Observability back-end (e.g Lightstep).

# OTel-specific
pip install opentelemetry-distro
pip install opentelemetry-exporter-otlp

# App-specific
pip install flask
pip install requests

A few noteworthy items:

Installing opentelemetry-distro will install a number of other dependent packages for instrumenting code, including opentelemetry-api and opentelemetry-sdk, and our auto-instrumentation wrapper binary, opentelemetry-instrument.
The opentelemetry-exporter-otlp package is used to send OTel data to your Observability back-end (e.g. Lightstep). Installing it in turn installs opentelemetry-exporter-otlp-proto-grpc (send data via gRPC) and opentelemetry-exporter-otlp-proto-http (send data via HTTP).

3- Install auto-instrumentation

As you may recall from earlier in this post, Python auto-instrumentation includes a binary that wraps our Python application and automagically adds some high-level instrumentation for us. But that's only part of the picture. There are Python auto-instrumentation libraries available for a number of popular Python libraries (e.g. Flask, requests). Using these auto-instrumentation libraries, along with opentelemetry-instrument, gives us auto-instrumentation superpowers. 💪

So how do we install these auto-instrumentation libraries? Well, there's a handy little tool for that, called opentelemetry-bootstrap. It was installed as part of our installation of opentelemetry-distro.

Let's run it:

opentelemetry-bootstrap -a install

So what does this do? The above command will read through the packages installed in your active site-packages folder, and will install the applicable auto-instrumentation libraries. For example, if you already installed the flask and requests packages (as we did in Step 2), running opentelemetry-bootstrap -a install will install opentelemetry-instrumentation-flask and opentelemetry-instrumentation-requests for you. If you leave out -a install, it will simply list out the recommended auto-instrumentation packages to be installed.

For more information on opentelemetry-bootstrap, check out the official OpenTelemetry docs.

4- Run the app

Here’s where it gets interesting! Normally to run this app, we’d run it like this:

python server.py

But if we did that, we wouldn’t be sending any OTel data to Lightstep. So we must instead do this:

export OTEL_EXPORTER_OTLP_TRACES_HEADERS="lightstep-access-token=<LS_ACCESS_TOKEN>"

opentelemetry-instrument \
           --traces_exporter console,otlp_proto_grpc \
           --metrics_exporter console,otlp_proto_grpc \
           --service_name test-py-auto-otlp-grpc-server \
           --exporter_otlp_endpoint "ingest.lightstep.com:443" \
           python server.py

Some noteworthy items:

Replace <LS_ACCESS_TOKEN> with your own Lightstep Access Token.
traces_exporter and metrics_exporter specify which trace exporter and which metrics to use, respectively. In this case, traces and metrics are being exported to console (stdout) and to otlp_proto_grpc. The otlp_proto_grpc option tells opentelemetry-instrument to send it to an endpoint that accepts OTLP via gRPC. The full list of available options for traces_exporter can be found here.
service_name sets the name of the service. This is the value that will show up in the Lightstep service explorer. Be sure to replace <service_name> with your own service name.
exporter_otlp_endpoint tells opentelemetry-instrument to send the traces to gRPC endpoint ingest.lightstep.com:443 (i.e. Lightstep).

Sample output:

Want to use HTTP instead of gRPC? First, you need to make sure that the pip package opentelemetry-exporter-otlp-proto-http is installed (should be automagically installed as part of installing opentelemetry-exporter-otlp).

Next, your opentelemetry-instrument command would look like this:

opentelemetry-instrument \
  --traces_exporter console,otlp_proto_http \
  --metrics_exporter console \
  --service_name test-py-auto-otlp-server \
  --exporter_otlp_traces_endpoint "https://ingest.lightstep.com:443/traces/otlp/v0.9" \
  python server.py

Some noteworthy items:

The traces_exporter uses otlp_proto_http instead of otlp_proto_grpc.
The exporter_otlp_traces_endpoint endpoint is https://ingest.lightstep.com/traces/otlp/v0.9 (see docs, instead of ingest.lightstep.com:443).
There is currently no metrics support for otlp_proto_http and there is no exporter_otlp_metrics_endpoint option, which is why metrics are being sent to console only.

5- Call the /rolldice service

Open up a new terminal window, and run the following:

curl http://localhost:8082/rolldice

Running the above line will return a random number between 1 and 6. Nothing too remarkable there. But if you look over at the terminal window for server.py, you’ll notice something in the output:

We see the trace from server.py! Why are we seeing this here? Because we set the --traces_exporter flag to console,otlp_proto_grpc, which exports to Lightstep via OTLP and to the console.

6- See it in Lightstep

OpenTelemetry Collector

The next approach to sending data to an Observability back-end is by way of the OpenTelemetry Collector. For non-development setups, this is the recommended approach to send OpenTelemetry data to your Observability back-end.

Sending OTel data via the OTel Collector is almost identical to what we did in the Direct from Application example above. The only difference is that:

We need to run an OTel Collector
When we run opentelemetry-instrument, our options are slightly different

Let’s look at this in greater detail below.

1- Follow Steps 1-3 from the “Direct from Application” example

2- Run the Collector

First, we need to configure our Collector for sending data to Lightstep. We do this by grabbing collector.yaml from Lightstep’s opentelemetry-examples repo.

git clone git@github.com:lightstep/opentelemetry-examples.git

Open up a new terminal window. First, you'll need to edit the collector.yaml file. Be sure to replace ${LIGHTSTEP_ACCESS_TOKEN} with your own Lightstep Access Token.

Now you can start up the Collector:

cd opentelemetry-examples/collector/vanilla
docker run -it --rm -p 4317:4317 -p 4318:4318 \
    -v $(pwd)/collector.yaml:/otel-config.yaml \
    --name otelcol otel/opentelemetry-collector-contrib:0.53.0  \
    "/otelcol-contrib" \
    "--config=otel-config.yaml"

Sample output:

3- Run the app

opentelemetry-instrument \
   --traces_exporter console,otlp \
   --metrics_exporter console,otlp \
   --service_name test-py-auto-collector-server \
   python server.py

Notice that the endpoint isn't specified. That's because it assumes that you are using the default Collector gRPC endpoint, 0.0.0.0:4317. The above command is the equivalent of saying:

opentelemetry-instrument \
  --traces_exporter console,otlp \
  --metrics_exporter console,otlp\
  --service_name test-py-auto-collector-server \
  --exporter_otlp_endpoint "0.0.0.0:4317" \
  --exporter_otlp_insecure true \
  python server.py

If you specify the endpoint, you must also specify --exporter_otlp_insecure true if a certificate isn't configured with your Collector.

Some additional noteworthy items:

otlp, used in configuring traces_exporter and metrics_exporter, is equivalent to using otlp_proto_grpc
To use a different Collector endpoint, simply replace it with your own. If you don't have a Certificate configured with your Collector, remember to add --exporter_otlp_insecure true
You don't need to set OTEL_EXPORTER_OTLP_TRACES_HEADERS, because that's already configured in the Collector's config.yml file.

If you wish to use HTTP instead of gRPC, the command would then look like this:

opentelemetry-instrument \
  --traces_exporter console,otlp_proto_http \
  --metrics_exporter console,otlp_proto_http \
  --service_name test-py-auto-collector-server \
  python server.py

Which is the same as saying:

opentelemetry-instrument \
  --traces_exporter console,otlp_proto_http \
  --metrics_exporter console,otlp_proto_http \
  --service_name test-py-auto-collector-server \
  --exporter_otlp_endpoint "http://0.0.0.0:4318" \
  --exporter_otlp_insecure true \
  python server.py

Again, if you wish to use your own Collector endpoint, simply replace the value in exporter_otlp_endpoint, making sure that you prefix it with http:// or https://. Remember to add --exporter_otlp_insecure true if you don't have a Certificate configured with your Collector.

Okay. Enough banter. Let's look at the sample output:

4- Call the /rolldice service

Open up a new terminal window, and run the following:

curl http://localhost:8082/rolldice

Sample output:

Again, we see the trace for server.py because we set the --traces_exporter flag to console,otlp, which exports to the Collector via OTLP and to the console.

5- See it in Lightstep

Launcher

If you thought it was easy-peasey to send OTel data to Lightstep à la auto-instrumentation binary, then it’s even easier to do it via the OTel Python Launcher! Think of it as an OTel wrapper to make it extra-easy to send data to Lightstep, by having a bunch of things pre-configured for you to lower that barrier to entry.

Sending OTel data via the Launcher is almost identical to what we did in the Direct from Application example above, with a few minor differences:

We have fewer packages to (yay!)
When we run opentelemetry-instrument, our options are slightly different

Let’s see it in action shall we?

1- Follow Steps 1-3 from the “Direct from Application” example

Minor change: replace the libraries from Step 2 with these:

# OTel-specific
pip install opentelemetry-launcher
pip install protobuf==3.20.1

# App-specific
pip install requests
pip install flask

We need to force a specific version of protobuf because of Launcher compatibility issues with newer versions. This was already fixed in opentelemetry-python.

When we install the opentelemetry-launcher package, it also does double-duty and doesn’t require that we run opentelemetry-bootstrap -a install.

2- Run the app

Be sure to replace <LS_ACCESS_TOKEN> with your own Lightstep Access Token.

export LS_ACCESS_TOKEN="<LS_ACCESS_TOKEN>"

opentelemetry-instrument \
    --service_name test-py-auto-launcher-server \
    python server.py

Looks like we have fewer options, don’t we? Let's dig in a bit to some noteworthy items:

We don’t need to specify an --exporter_otlp_traces_endpoint, because that’s already implicitly done for us, and as set to ingest.lightstep.com:443.
Instead of setting a messy-looking environment var for our Lightstep Access Token (export OTEL_EXPORTER_OTLP_TRACES_HEADERS="lightstep-access-token=<LS_ACCESS_TOKEN>"), we just have to do this: export LS_ACCESS_TOKEN="<LS_ACCESS_TOKEN>", which looks way cleaner.

If you wish to send your OTel data via a Collector instance first, rather than direct from your application, you would do this instead:

opentelemetry-instrument \
    --service_name test-py-auto-launcher-server \
    --exporter_otlp_traces_endpoint "0.0.0.0:4317" \
    --exporter_otlp_traces_insecure true \
    python server.py

Noteworthy items:

Do not set LS_ACCESS_TOKEN, since that's already configured in the Collector's config.yml file.
If you attempt to override exporter_otlp_endpoint to send traces to a Collector, the traces will be sent directly to ingest.lightstep.com:443 instead of via the Collector. Instead, you need to override exporter_otlp_traces_endpoint,
exporter_otlp_traces_endpoint sends traces to a Collector running on 0.0.0.0:4317 (gRPC). If you wish to use a different Collector address, simply include exporter_otlp_traces_endpoint, using your own Collector's endpoint.
exporter_otlp_traces_insecure is set to true. This is required if you are using a Collector and if a certificate isn't configured in the Collector.
There is currently no HTTP support for Python Launchers.

Sample output:

3- Call the /rolldice service

Open up a new terminal window, and run the following:

curl http://localhost:8082/rolldice

Sample output:

Notice that since our opentelemetry-instrument call didn't specify a --traces_exporter, it's the equivalent of saying --traces_exporter otlp_proto_grpc. I also means that there's no trace output to the console (stdout).

4- See it in Lightstep

Should I always use the auto-instrumentation binary?

Is opentelemetry-instrument still helpful even if you’re not using a Python library that’s not auto-instrumented? Personally, I think so! Consider this file, client.py:

from sys import argv

from requests import get

from opentelemetry import trace
from opentelemetry.propagate import inject
tracer = trace.get_tracer_provider().get_tracer(__name__)

assert len(argv) == 2

with tracer.start_as_current_span("client"):

   with tracer.start_as_current_span("client-server"):
       headers = {}
       inject(headers)
       requested = get(
           "http://localhost:8082/rolldice",
           params={"param": argv[1]},
           headers=headers,
       )

       assert requested.status_code == 200

Let’s run the above program with the auto-instrumentation binary. Be sure to replace <LS_ACCESS_TOKEN> with your own Lightstep Access Token.

export OTEL_EXPORTER_OTLP_TRACES_HEADERS="lightstep-access-token=<LS_ACCESS_TOKEN>"

opentelemetry-instrument \
    --traces_exporter console,otlp \
    --service_name test-py-auto-client \
    --exporter_otlp_endpoint "ingest.lightstep.com:443" \
    python client.py test

Notice that aside from creating spans in client.py, there’s no OTel configuration in there. You don’t configure the service name, the exporter, or the endpoint. That’s all taken care of when you run opentelemetry-instrument. Plus, if your code happens to use a library that is auto-instrumented, you don’t have to do anything else.

Note: If you’re wondering why we’re executing the command python client.py test, it’s because client.py takes a single parameter, which in this case is called test.

gRPC Debugging

Do you ever wonder if your gRPC calls are going into a black hole? I definitely do! When I was mucking around with gRPC for the Golang OTel libraries, I learned about some gRPC debug flags that would make my life easier for troubleshooting gRPC connectivity issues. Which of course got me wondering if there was a Python equivalent. Turns out there is. Set these environment variables before running your app, and you’re golden:

export GRPC_VERBOSITY=debug
export GRPC_TRACE=http,call_error,connectivity_state

This means that when we start up our server.py, we get something like this:

And then when we call our endpoint via curl, we get this:

The part highlighted above tells me that our gRPC call was successful!

Final Thoughts

Auto-instrumentation in Python is pretty freaking awesome, and it really lowers the barrier to entry for OpenTelemetry. As we saw with the Direct from Application and Collector examples, the code stays pretty much the same. The only difference is that you need to change up some flags so that the auto-instrumentation binary knows where to send your traces to. Nice and easy!

In case you’re wondering, there is a totally pure OTel Python manual instrumentation approach, which I will cover in a future blog post, so stay tuned! For now, bask in the fact that you learned something super cool today about OTel Python auto-instrumentation!

And now, I will reward you with a picture of my rat Phoebe getting some cuddles.

Peace, love, and code. 🦄 🌈 💫

Got questions about OTel instrumentation with Python? Talk to me! Feel free to connect through e-mail, or hit me up on Twitter or LinkedIn. Hope to hear from y’all!

Observability Mythbusters: Yes, Observability-Landscape-as-Code is a Thing

Adriana Villela — Fri, 19 Aug 2022 15:51:54 +0000

You might be thinking that this might be yet another overloaded term to add to your Buzzword Bingo lineup. Maybe you’re rolling your eyes at me over Yet Another X-as-Code thing. Or maybe you want to hurl your shoe at your screen in frustration. But aren’t you the teeeeniest bit curious? I invite you to stick around to satisfy your curiosity before you pass judgment.

Take my (virtual) hand, and let’s jump in!

Observability-Landscape-as-Code?? Wut?

Observability is about good practices. Good practices are useless unless you have a consistent landscape. In order to support these practices, there are a number of setup-y-type things that are required to enable teams to truly unlock Observability’s powers.

With so many integrations and moving parts, it can be hard to keep track of all the things that you need in order to achieve Observability greatness. This is where Observability-Landscape-as-Code (OLaC) can help. OLaC means supporting Observability by codifying your Observability landscape to ensure:

Consistency ✅
Maintainability ✅
Reproducibility ✅

Our Observability Landscape is made up of the following:

Application instrumentation
Collecting and storing application telemetry
An Observability back-end
A set of meaningful SLOs
An Incident Response system for alerting on-call Engineers

Observability-Landscape-as-Code makes this possible, through the following practices:

Instrumenting your code with OpenTelemetry
Codifying the deployment of the OTel Collector
Using a Terraform Provider to configure your Observability back-end
Codifying SLOs using the vendor-neutral OpenSLO specification
Using APIs to configure your Incident Management systems

Let’s dig a little deeper.

Instrument Your Code

The most logical place to start when defining your Observability landscape is with instrumenting your application. After all, without instrumentation, you can’t send anything to your Observability back-end, which means that there is no way in Space that your system can be Observable.

OpenTelemetry (OTel), the vendor-neutral open-source Observability framework, is the means by which applications are instrumented. OTel allows you to instrument code written in a number of major programming languages.

But…what exactly should you instrument? According to Ted Young, you should “add tracing to your frameworks and libraries, and try to keep it out of your application code as much as possible.”

But whyyyy??

Let’s break it down.

An application is typically made up of a combination of frameworks and libraries. Think of your application code as the glue that holds these frameworks and libraries together. The frameworks and libraries being used in an application can be a combination of home-grown and third-party offerings. If you instrument your home-grown frameworks and libraries, then you have all the coverage you need, as far as tracing is concerned.

Yes, technically you could add Spans to your application code, but chances are you’ll get caught up copying and pasting Spans all over the place. And next thing you know, you’ve polluted your code with unnecessary Spans. Plus, if you decide somewhere down the line that you want to change up how you’re creating your Spans, chances are you’ll end up having to do some search and replace to fix stuff up, which all know can be messyyyyy.

That said, it is still useful to have your application code add Events and Span Attributes which are specific to that particular operation and would add value. This is how you model the actual business logic of your service! You just have to be selective.

As Ted cautions, “Spans are a structural element and it's usually a smell if they end up mixed in with application code. In general, any instrumentation which gets repeated across many instances of application code, such as adding a **project_id as an attribute, should get moved into a framework hook or a shared function of some kind.”

Eeeee! That sounds a bit stressful, doesn’t it. And let’s face it…it’s a mighty tall order to tell developers to instrument all their framework and library (and sometimes application) code, but it’s the only way that this Observability thing will really do right by you. But, DON’T PANIC. Here’s what you do:

If you’re instrumenting previously-uninstrumented code, focus on instrumenting your most important services first, so that it feels less overwhelming.
When working with net-new code, follow Observability-Driven Development (ODD), whereby you instrument-as-you-code, instrumenting your code ends up being waaaay less overwhelming. Whew!

The idea is to add enough Traces and supporting data (including Attributes, Span Events and Metrics) to your code, such that it can send enough data to your Observability back-end to allow developers and operators to troubleshoot issues quickly and effectively. Remember, we want to help them answer that nagging question of “Why is this happening?”

Use the OTel Collector & Codify its Deployment

Congrats! You’ve instrumented your code! Your Observability Landscape is looking slick AF! Now, while OTel’s instrumentation libraries are used to instrument application code, we need another OTel goodie as part of our Landscape: the OTel Collector. The Collector is a vendor-neutral agent that ingests, transforms, and exports data to one or more Observability back-ends.

And in case you’re wondering, yes, you could technically send data directly from your instrumented code to your desired Observability back-end sans Collector; however, this isn’t the best way to go about it. The Collector serves as a data pipeline which allows you to do things like mask, filter, transform, and append data before sending it to your Observability back-end. It can also ingest Metrics data from your infrastructure, like your Prometheus-style metrics generated by your Hashi stack or Kubernetes cluster metrics.

Now, if you’re adding an OTel Collector to the mix, then it means that it needs to run somewhere. That somewhere can include Kubernetes, Nomad, or Virtual Machines (VMs) as a binary. Regardless of where you deploy it, you will need some sort of Infrastructure-as-Code (IaC) thing to configure and deploy the Collector. Luckily, you can use IaC tools like trusty ‘ole Terraform, superfly competitor Pulumi, oft-forgotten-yet-still-cool-kid Ansible. Or heck, write your own custom scripts to do it. It’s up to you. Just. Codify. It.

Codify Observability Back-End Configuration

Now we’ve covered application instrumentation and data transmission. Next we must tackle The Thing that Receives the Data. That is, your Observability back-end. Now, as with many-a-tool out there, Observability back-ends require some tweaking so that they work in a way that makes them useful to us. Such configurations can include creating (non-sucky, meaningful) dashboards, Slack integrations, or saved queries for your Traces.

And wouldn’t you know it…it turns out that most of the cool kids in the Observability back-end game have Terraform Providers that facilitate said configurations, including Lightstep. If you don’t believe me, go to the Terraform Providers search bar and enter an Observability back-end name. I dare you. MUAHAHAHAHAHA…

Note: As far as I know, there are no equivalents from Pulumi and Ansible (wink wink, nudge nudge), but please correct me if I’m wrong!

Regardless of what Observability back-end you use, make sure that you use the heck out of your Observability back-end’s Terraform Provider to codify as much of that setup stuff as possible. That way, if you happen to bork your setup, you can always go back to your Terraform code. And remember: make sure that you resist the temptation to make tweaks to things on the UI that are supported by your Provider.

Get your SLO game on, and Codify!

If we take a step back and admire our Landscape, you’ll see that it now includes application instrumentation, sending data to our Observability back-end, and setting up our Observability back-end. And while we’re on our Observability back-end, let’s talk Service-Level Objectives (SLOs).

If you’re an on-call Engineer, chances are you don’t want to get woken up by just any alert. If you get woken up, it had better be for a damn good reason. I once got woken up at 2am over a development environment alert by the Network Operations Center (NOC). Not fun. And not needed. Aaaand…because I was totally wired after being woken up, I had a really hard time getting back to sleep after that. Nobody wants to deal with unnecessary alerts.

Alerts are usually triggered when a certain threshold is reached. For example: low disk space, high CPU, high RAM. There will always be a need for alerts, but if we have too many alerts, it becomes overwhelming, and it becomes hard to tell what’s important and what’s not.

Good Observability practices dictate that we should only set up meaningful alerts. Defining a small, yet meaningful number of alerts is a good thing, because it means that the alerts that you do get are the really, really important ones, and are less likely to get ignored as “white noise” or worse, cause wake you up when it wasn’t wake-up-worthy. Service-Level Objectives (SLOs) can help with this, by helping us identify what data ingested by your Observability back-end are important and alert-worthy. SLOs connect telemetry with specific customer experiences, so that the entire organization can understand the relationship between the software system and the business goals.

If you really want to up your SLO game (and you definitely should), you need to define your SLOs as code. This is where OpenSLO comes to the rescue! Originally started up by Nobl9, OpenSLO is an open-source “specification for defining SLOs to enable a common, vendor–agnostic approach to tracking and interfacing with SLOs.” Super dope. Don’t you just love standardization? 💜

Note: Although it’s outside of the scope of this post to dig deep into this topic, in case you’re curious, you can check out what an OpenSLO YAML definition looks like here.

Use APIs for Incident Response Tool Configuration

As we saw above, SLOs trigger alerts that are actioned by on-call Engineers. These on-call Engineers are made aware of said alerts, courtesy of an Incident Response (IR) tool. Like our Observability back-end, IR tools require a bunch of configuration in order for them to be useful to us and to our teams.

Things you need to configure:

On-call team and contact information
On-call schedule
Call tree
Triggers (i.e. things that cause you go get paged)

Luckily, many IR tools have APIs that let you set up and configure them, in lieu of clickity-clicking through a UI. Choosing an incident management system with a robust API that lets you do all of the above through code ensures that there are no knowledge bottlenecks, as everyone in your team has access to config as code via a Git repo.

To put things into perspective, I will tell you a little story. At one of my previous workplaces, we used an incident management system that was beyond annoying to configure and understand. I swear that it was a relic of the Web of the early 2000s. There was no API. There was one guy who knew how it all worked, and luckily I made friends with him, because there is no way that I would’ve known how to configure the on-call schedule for my team. Unfortunately, he was the only guy who knew how the system worked, so if he was on vacation, and you needed to onboard a new team or team members, you were hosed.

Putting it all together

Today we learned that OLaC isn’t just some bogus, buzzwordy, cringe-worthy term, but in fact refers to supporting Observability by codifying your Observability Landscape, thereby making it consistent, repeatable, and maintainable.

Observability-Landscape-as-Code makes this possible, through the following practices:

Instrumenting your code with OpenTelemetry, following ODD to instrument your code as you go. ✅
*Codifying the deployment of the OTel Collector *(to Nomad, Kubernetes, or a VM) using tools such as Terraform, Pulumi, or Ansible. The Collector funnels your OTel data to your Observability back-end. ✅
Using a Terraform Provider to configure your Observability back-end. Fortunately, most Observability back-ends, like Lighstep, have their own Providers. ✅
Codifying SLOs using the vendor-neutral OpenSLO specification. SLOs are important because they are the driving force behind alerts. SLO-based alerts ensure that on-call Engineers get meaningful, data-driven alerts. ✅
Using APIs to configure your Incident Management systems, which are usually the conduit for paging on-call Engineers. Many incident management systems have APIs that can be used to configure the platform for your organization’s specific needs. This includes things like managing incidents and account settings. ✅

That, my friends, is how you have a kick-ass Observability Landscape.

Congrats! You made it! I will now reward you with a picture of my dearly departed rat, Susie.

Peace, love, and code. 🦄 🌈 💫

If you’d like to share stories of your Observability journey, or just ask Observability-related questions, hit me up on the Lightstep Community Discord, or get in touch by e-mail. Hope to hear from y’all!

Leading SRE with Empathy

Ana Margarita Medina 👩🏽‍💻 — Mon, 15 Aug 2022 21:55:00 +0000

Writing and operating software is hard.

We've seen huge changes in how we build reliable software over the past two decades, driven by the DevOps movement and SRE practices the last few years. Some of the great things that came from the years of experience and great minds in the community are that operating software at scale is very hard, and it’s not just about code and processes. We have learned that people are also a big part of the puzzle. High-performing organizations and teams make sure to prioritize people. We need a lot more of that; it’s common for DevOps and SRE teams to burn out and suffer from pager fatigue or even on-call PTSD.

Today, I’m only focusing on SREs. What if we train SREs to lead with empathy? It’s not just about keeping servers healthy and green and our users and CTOs happy. But what is empathy? While empathy may be defined as this definition, I think empathy is the ability to relate to other human beings by being curious, listening, offering help while building trust.

First, what is SRE?

SRE stands for Site Reliability Engineering. This practice has been around for a while, but with different names like Operations Engineering, Systems Engineering, or Production Engineering through the years. SRE got more established when Google released the book on SRE, which I refer to as the SRE bible. You can read it for free online via this website.

An SRE is an engineer that focuses on system reliability. Some folks like to think of SRE as engineers that spend 50% of their time writing code and the other 50% of their time operating and optimizing software for the purpose of reliability. When we talk about SRE, we often get asked how SRE and DevOps vary from one another. The best way I’ve seen it explained is covered on Google Cloud’s “What's the Difference Between DevOps and SRE?” with Liz Fong-Jones and Seth Vargo.

SRE is an implementation of DevOps.

It’s DevOps with the purpose of reliability.

What does an SRE do?

SRE duties vary from company to company. Some of the duties one can find themselves doing while being an SRE include (and are not limited to):

Writing code
Being on-call
Setting up tooling (CI/CD, Observability, Incident Response tooling, Testing pipelines, etc.)
Setting up alerting
Setting up SLOs
Capacity Planning
Running Chaos Engineering Experiments
Advocating for modern and best practices
Running blameless postmortems
Reviewing the company’s testing practices
Auditing services
Interviewing other teams
Reading other company’s postmortems
Improving processes
Reviewing Architecture decisions
Drafting new architecture plans or PRDs (Product Requirements Documents) on vendor/tooling changes

When we talk to seasoned SREs, we learn that their empathy has allowed them to be more successful. This is because, as an SRE, you have empathy for the service teams one supports and understands that they are doing their work to the best of their ability. But it’s not just them; this also expands to the other SREs in one’s organization. As I said above, SRE varies from company to company, and every SRE comes from a different background. But this also extends to other duties like educating the rest of the organization on reliability best practices.

How can SREs embody empathy?

Empathy goes a long way; being empathetic builds engineers' knowledge level and increases engineering morale and company culture. It begins with listening to the needs of the teams they support and the issues they are working on. Within their teams, this might mean being able to support one another when trading on-call shifts, triaging bugs, dealing with incidents, or leading incident review meetings.

Building empathy takes practice, and every SRE might embody it differently. For some folks, it might look like this:

curiosity
patience
open communication
ability to admit being wrong

SREs should stay curious and ask as many questions when reviewing documentation, incidents, or data. This allows one to try to get the most context into decisions and understand the “Why.” Curiosity lets an SRE embark on their continuous learning journey.

We know that the work SREs do can be very time-demanding and stressful; the ability to stay calm under pressure is sometimes one of the best qualities of an SRE. As people are the ones that operate our systems, one needs to understand that the full context of a system might not be seen through a dashboard, and one must be patient to research what exactly happened and debug a bit further. We also see that when an SRE practice is still in its early days, you will need a lot of patience to work with leadership and other teams for the organization to prioritize reliability action items every sprint.

Additionally, like any work or personal relationship, the more one can communicate, the better; providing data with this is helpful, especially when dealing with incidents. Over-communication is appreciated and goes a long way. Let’s take, for example, when we are dealing with incidents. When Jewel sees that their system is starting to degrade its performance, they should say something. Jewel can say something and start researching on their own, but this gives the heads up to the rest of the team that something could be happening, and it also opens the door for another team to share any insights they might have. It’s common for two people to be working on the same system at the same time. In another scenario, let’s take an ongoing incident. There is a Slack channel set up and a Zoom call ongoing; engineers are researching, debugging, and trying to bring a system to a better state. It’s good practice to bring in as much relevant data from other sources that provides more context into why the system might have left its unhealthy state. With communication practices like this, one might be able to resolve an incident in less time or just be able to provide more context when a postmortem is written up.

As we think about building distributed systems across distributed teams, we see that over-communication helps build empathy. This helps build trust and allows room for failure. After all, we are humans working on complex systems. Additionally, the ability to be vulnerable, open, honest, and transparent with our peers is important. Our systems are changing rapidly, and our mental models will be outdated. Cultivating a culture where failure is embraced and a culture of experimentation is cultivated is important and essential. This will also be very helpful for an honest postmortem. One gets to show up for their team.

As an SRE, leading with empathy means that you become an individual that has established a lot of trust in the organization. This has several benefits:

Teams might ask you earlier in the development processes/issue process for feedback on the issues they are facing (ability to approach)
Discourse at incident reviews is less about blaming others and more about learning and sharing and moving forward together (trust)

What world are we creating with more empathy?

It is common to find empathy in SRE and DevOps communities, to the extent that there is a huge movement behind #HugOps. For all of our favorite companies, applications, and tools, there’s always a team of folks working hard to keep things running or firefighting when things go wrong. When those sites and applications go down, we acknowledge the outage and downtime sucks, but we send our condolences and support to those working to bring things back. We know the work is hard, and we stand together in support as a community. Hugs are sent just like packets across the internet to our friends.

_We must continue cultivating this culture and ensuring that all engineers follow it. _

Disclaimer: Emotional SRE Toil

In SRE, we talk about toil. Toil is the manual work needed to keep systems running in production. Toil is defined as “the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows.” When we think about our SREs embodying more empathy, we need to look at the other side of the coin. We are asking our teammates to take on a larger work that can not be quantified: emotional work.

Understanding other people and being in someone else’s shoes eventually catches up to you.
We already know SREs sometimes struggle with mental health, which is worth mentioning in these discussions. How do we get ahead of the possible emotional toil that comes from being more empathetic? We first need to accept that this is already happening in our organizations, but we are not talking about it.

Looking at the workload of SREs during the pandemic and # of incidents that they fire fought or the number of services they helped in, we can see that they’ve been a massive help to our organizations and keeping our users happy. How do we protect the humans that support our systems and keep our services up?

This is a call to action for our leaders to check in with their teams more. This entails listening to quarterly retrospectives, attending post-mortem reviews, and revising the information we collect about our services, requests, and incidents. This also includes other manager-y duties like listening to ideas your team brings to the table, proactively tracking and ensuring your team takes time off (I’m looking at you managers at unlimited PTO companies) along with ensuring that on-call schedules are balanced across the organization, ensuring your SMEs (Subject Matter Experts) are not very close from burning out, and ensuring that other people are also getting trained to be on-call.

To make an impact, our SRE leaders need to lead with empathy and help the rest of the organization engineer with empathy.

Stay tuned as we continue the conversation and talk about empathy with our users next. Do you have any thoughts? Feel free to reach out via Twitter at @ana_m_medina or find me on LinkedIn.

This post was originally posted on the Lightstep Blog