Forem: Adriana Villela

The Humans of OpenTelemetry: KubeCon EU 2024

Adriana Villela — Mon, 17 Jun 2024 04:00:00 +0000

We’re back with our second edition of Humans of OpenTelemetry, this time from KubeCon EU in Paris. Once again, Reese Lee and I interviewed OpenTelemetry contributors and end users, and learned how they got involved with OTel:

Also, special thanks to:

Reese Lee, my co-interviewer
Henrik Rexed for providing the audio and video recording equipment, and doing the initial edits of the raw footage
Zhu Jiekun for assisting with his own camera

You can watch the full recording below:

The Humans of OTel - KubeCon EU 2024 - YouTube

We caught up with some of the contributors, maintainers, and practitioners of OpenTelemetry at the OpenTelemetry Observatory for KubeCon EU 2024 in Paris. Th...

youtube.com

Thanks to everyone who has contributed to OpenTelemetry to date, and we look forward to your continued contributions in 2024 and beyond! 🎉

Transcript

If reading is more your thing, check out the transcript of our conversations below.

1- Meet the Humans of OTel

IRIS DYRMISHI: Well, I’m Iris Dyrmishi. I’m a senior observability engineer at Miro and my life, my professional life is all about observability. I build an observability platform that provides the tools for engineering teams at Miro to monitor, to observe and get the best of their applications.

SEVERIN NEUMANN: My name is Severin Neumann. I’m working at Cisco at the open source program office and I’m a member of the OpenTelemetry governance committee and I’m one of the co-maintainers of the OpenTelemetry documentation.

KAYLA REOPELLE: My name is Kayla Reopelle. I work for New Relic and I am contributing to the OpenTelemetry Ruby project.

MORGAN MCLEAN: My name is Morgan McLean,I’m a director of product management at Splunk.I’ve been with OpenTelemetry since day one. I’m one of the co-founders of the project. I’m on the governance committee. Wow. What do I work on within OTel?A bit of everything. I mean early on it was literally everything. Myself and Ted and various others were doing many, many jobs. More recently I was involved in the release of traces, metrics 1.0. Logs 1.0 last year. Right now I’m working on profiling as well as OpenTelemetry’s expansion into mainframe computing.

HENRIK REXED: My name is Henrik Rexed. I am a cloud native advocate at Dynatrace and I’m passionate about observability, performance, and I’m trying to help the community by providing content on getting started on any solutions out there.

VIJAY SAMUEL: My name is Vijay Samuel and I help do architecture for the observability platform at eBay.

DANIEL GOMEZ BLANCO: I’m Daniel Gomez Blanco. I’m a principal engineer at Skyscanner and also member of the OpenTelemetry governance committee.

DOUG ODEGAARD: My name is Doug Odegaard. I’m a senior solutions architect with ServiceNow Cloud Observability, which is also formerly Lightstep. I’m also a previous customer of using OpenTelemetry for several years prior to that.

ADNAN RAHIĆ: Hey, I am Adnan. I work at Tracetest as a developer advocate which is…you can guess better than me what that is. Pretty much do a bunch of everything regarding OpenTelemetry. I’m one of the contributors for the documentation, for the blog, and the demo.

RYNN MANCUSO: My name is Rynn Mancuso. I work for Honeycomb.io and I am one of the maintainers of the End User SIG.

2- What does observability mean to you?

IRIS DYRMISHI: What does observability mean to me? observability to me is the biggest passion of my life and also my professional career. It is one of those areas that you are not very interested when you start your career because you don’t know anything about it. It’s not taught in school, it’s not preached by the tech communities a lot, but then you discover it and say, “Wow, this is amazing!” We’re actually making a change and we’re helping the teams make the best of their product. So yeah, that’s all.

SEVERIN NEUMANN: I think observability is a big game changer, right? So it’s evolution from what we have done, especially APM, over the last few years. So I worked for a very long time at AppDynamics and we sold APM agents to customers and we gave them a lot of the things that observability is promising today as well. But the big change I see with observability that it’s coming down, let’s say to everybody, right? So this is making the things that we did there available for everybody. And even more, we’re moving away from this… Hey, let’s add a post compilation agent into your application to like, yeah, let’s make native observability. Let’s make this a thing that developers, that operation teams are using across all the organizations.

KAYLA REOPELLE: So to me observability means having peace of mind. It means having something that you can rely on in order to see what happened and what went wrong. I think observability is also a way to feel more technically connected to your customers and your users, so that you can see the ways that they’re interacting with your software instead of just the ways that you might interact with it.

MORGAN MCLEAN: I mean, observability to me transcends just the computing industry. It’s the ability to peer into something and understand how it works, what it’s doing right now, and thus if it breaks, how to fix it more quickly. Certainly when we think about telemetry in this industry, what observability classically has meant is visibility to backend infrastructure and applications kind of excitingly, I think it’s expanding right now, right? With OpenTelemetry we’re pushing into client applications, we’re pushing into mainframes, as I mentioned earlier. And so it’s really visibility into any systems that impact your business, any technical system observability.

HENRIK REXED: Usually when people mention of observability they say it’s a replacement of the old name monitoring. But in fact for me it’s more than monitoring, because monitoring is like, you just look at something and observability is like having enough information to understand a given situation. So if you just look at metrics then, okay, you have a guess that something is going on, but you don’t understand. So having the options to get more information like logs, events, exceptions, traces, compiling, then at the end combine all those dimensions together, then you say, okay, I got it, this is my problem and I can resolve it.

VIJAY SAMUEL: What does observability mean to me? I belong to what is called the site engineering organization inside of eBay, and our goal is to make sure that we can observe everything that’s going on in the site and ensure that we have high availability. So basically, observability means knowing if the site is running fine or not, because that’s why I’m there.

DANIEL GOMEZ BLANCO: What does observability mean to me? It’s a way for us to understand what’s happening within our systems, because we run quite a complex system, so we need to understand what goes on inside of them so we can deliver a good experience for our end users at the end of the day.

DOUG ODEGAARD: So observability is, to me, I’ve been a full stack developer for years, and so as we observe…actually I ended up on an incident response team doing tracking of incidents, but also trying to figure out what was wrong. And it pointed out to me how much we need this, how hard it was to look at so many different screens and so forth.

ADNAN RAHIĆ: So observability is, to me, I’ve been a full stack developer for years, and so as we observe…actually I ended up on an incident response team doing tracking of incidents, but also trying to figure out what was wrong. And it pointed out to me how much we need this, how hard it was to look at so many different screens and so forth.observability for me is the way to actually see what’s happening in your system. It’s the pinnacle of not being up the whole night trying to figure out what went wrong. And with OpenTelemetry and with the rise of tracing the last couple of years, it has hit an all time high with regards to the possibilities that we have right now. So I’m just really, really happy to be part of the project. I’m also really happy that it’s growing at that pace, that it’s growing right now, and I can’t see how that’s going to evolve within the next couple of years.

RYNN MANCUSO: For me, observability is about being able to ask deeper questions of our systems, being able to demand, I think more than just alerting on things that are emergencies, things we’ve seen before, but actually being able to go out into the unknown and understand how complex systems are performing.

3- What does OpenTelemetry mean to you?

IRIS DYRMISHI: OpenTelemetry is the tool that is making observability great again. I would say that observability is seeing the surge, now that OpenTelemetry is becoming so popular, it’s allowing centralization of telemetry signals, it’s allowing semantic conventions, and it’s generally helping observability teams and engineering teams take more attention to the observability and building it and making it better.

SEVERIN NEUMANN: What does OpenTelemetry mean to me? I think it’s the vehicle for observability. It’s enabling that. And I joined OpenTelemetry community a few years back because I was curious about this idea to bring observability to everybody. And I think we are doing a really good job. And what it also means to me now is that it’s an amazing community. Right? So we’re at KubeCon here, and I meet so many people I just know from those conversations, and now I can talk to them in person. And we talk a lot about OpenTelemetry, but we also talk a lot about other things than OpenTelemetry. We talk about observability, of course, about what we think about is going to happen in a few years and all those other things, and that’s what OpenTelemetry means to me.

KAYLA REOPELLE: So OpenTelemetry to me seems like it’s a community effort to take the best of what’s already been out there for instrumentation and collect it in one group so that everyone can benefit from it. I think that we’ve learned so much as different agent engineers, but there’s also so much to learn from users of the products themselves. And OpenTelemetry does a great job of bringing both people who are, you know, experts in observability, and experts in languages to make something that’s really great and meaningful for everyone.

MORGAN MCLEAN: I mean, OpenTelemetry is my baby. Put so much effort into creating this project. What does it mean to me? I mean, there’s the boring answer, which is it extracts signals: metrics, traces, logs, profiles, everything else from your infrastructure, from your services, from your clients, makes those observable, processable on the backend. But I think to a lot of us who’ve been in this community so long, and a lot of us like yourself and Henrik here and others who participate in the community so much, I mean, OTel is also just a really nice open source community to participate in. It’s a thing I just enjoy working on. I know that’s abstract and kind of like a sort of squishy thing to say, but I don’t know. OTel has a lot of meaning to me in many different ways. All very positive.

HENRIK REXED: OpenTelemetry for me, means the future. Because at the end, by having an open standard, we have the luxury to have a common standard for common format, for all the solution of the market and having that common format for all the industry and all the vendors and all the solutions, it will just open use cases. I think testing used to rely on, I don’t know, feedback from users. And now with observability data, we could be so much efficient in the way we’re testing, we could be so much efficient in replacing marketing tools, business analytics tools. I think it’s the future. And one thing that also a lot of people talk about, AI everywhere, machine learning, blah, blah, blah, but I think it’s the same thing as a Tesla. I mean, Tesla, when you drive your car, it takes decisions based on the sensors that it measures. And if you don’t have those sensors and those measurements, then you cannot have a smart… you can have the smartest systems, but without the data, you cannot take the right decisions. I think it’s an enabler also for the future implementations of modern applications.

VIJAY SAMUEL: OpenTelemetry is the standard for observability going forward, and it’s very important because as we have gone through the journey of observability over the past few years, we have had to hunt for open standards in Prometheus and few others. Now, at least with ingestion and collection, it’s a single standard for everyone to adopt. And I think that’s pretty powerful for the long run.

DANIEL GOMEZ BLANCO: What does OpenTelemetry mean to me? That, I think is bringing people together, bringing everyone together under one single language and the ones that way of thinking about telemetry. I think human languages are difficult enough for us to understand each other. And I think, you know, OpenTelemetry is bringing the technology together and one single way of like, thinking about telemetry, thinking about how we observe our systems.

DOUG ODEGAARD: To me, OpenTelemetry is bringing the ability to have product teams, infrastructure teams, helping their jobs make it easier and also just improve the customer experience and just make it overall a better experience to do our jobs.

ADNAN RAHIĆ: OpenTelemetry is the, I’m going to say, the future of observability. We’ve seen so many companies, many vendors move to an OpenTelemetry-first mindset, and the way that you can use OpenTelemetry to generate them, to actually gather all telemetry signals with one set of libraries, with one tool. It’s just the way it was supposed to be. You’re not locked into one tool, one vendor, one cloud provider anymore. You can do basically whatever you want, and you can use both the metrics, logs, and traces for basically anything you want to do. Really happy to see it.

RYNN MANCUSO: OpenTelemetry is an instrumentation protocol that helps us ask more detailed questions about observability because it collects multiple signals from many flexible types of systems. Folks monitor everything from the control plane in Kubernetes all the way up to physical on-prem systems. It’s a really flexible language and it’s beautiful community of humans that came together over the pandemic to build something really special.

4- How did you get involved with OpenTelemetry?

IRIS DYRMISHI: I was working in a very fast-pacing observability team, and we were maintaining a lot of tools and we really did not have conventions there, we did not have centralizations and we really were not flexible when it came to backends and vendor agnostic in general. So we discovered this amazing tool called OpenTelemetry. We said okay, let’s give it a try. It worked great for us. And here I am today, one year later, more than one year later, and let’s say pushing the migration to OpenTelemetry in my second project.

SEVERIN NEUMANN: How did I get involved into OpenTelemetry? So yeah, I mentioned that… so I got curious a few years ago. So I was… I was at AppDynamics working as a so-called domain architect, and I was an expert for Node.js, Python and a lot of those other languages. And there was always this conversation around like, hey, there’s this thing now called OpenTelemetry and should we not integrate this into our product? And I was like, okay, I want to learn more. Then I was like, what is a good way to learn something new about an open source technology? Yeah, get involved into that. So I was involved in JavaScript at some point, and then at some point I realized like, yeah, but if I really want to get a good view into OpenTelemetry, doing documentation is a good way into that. And that’s how I ended up being a maintainer for the documentation.

KAYLA REOPELLE: I got involved in OpenTelemetry last spring when New Relic asked me to take a look at what the current status was of the OpenTelemetry Ruby project. I also work as an engineer on the New Relic Ruby agent team, and that gave me an opportunity to start to contribute to the project. And I noticed that a lot of the signals for Ruby weren’t yet stable. So a lot of my work so far has been going into trying to bring logs and metrics to stability in Ruby.

MORGAN MCLEAN: I was working at Google on Google’s observability products like tracing, profiling, debugging, that sort of thing. And one of the challenges we had in tracing was getting data from people’s applications. It’s really, really hard. You need integrations of hundreds of thousands of pieces of software. No one team, no one company is going to maintain that. It’s just infeasible. And so we want to do something open source. There were other open source standards. There was one that had started, I think, roughly around the same time we were doing this, called OpenTracing. We started OpenCensus.

At some point, especially amongst the more social media savvy members, the team, which I am not one of, there was some contention between those projects about where people who maintain databases and language runtime things should actually spend their integration efforts, and it was limiting the success of both projects. So I was leading OpenCensus. Ted and Dan and others were leading OpenTracing. And in late 2018, early 2019, we finally sort of brought things to a head and decided to merge those into what is now called OpenTelemetry. So that’s sort of, you know, I’ve been involved since then, I’ve been…now I work at Splunk. Different company, but still on the same types of things. But that’s how my involvement started, and it’s just grown and grown and snowballed from there.

HENRIK REXED: When I started the adventure in observability, of course, I joined Dynatrace, and Dynatrace has their vendor agent, the OneAgent, and I saw this movement of OpenTelemetry, and coming from the performance background, I looked at it and I said, “Whoa, an open standard.” “That sounds quite exciting” because I had a performance, a gig for a customer, where I implemented like a collecting logs and processing it and putting machine learning. And I told myself at that time, it would be so wonderful to have one common standard. So then instead of doing a custom implementation, I could have something for everyone. And when I looked at the, just the definition of the project and the things behind the project, I was so excited. I said, oh, gosh, I want to be involved in the project. And that’s where I started to build content to help the community get started.

I used to be a developer, but I’m a bad developer for sure. So that’s why I’m trying to help the project in other ways, in all the directions. And yeah, my goal is increase the adoption of the open standards, making sure that it’s been adopted everywhere, so then we can move forward by implementing even more exciting implementations.

VIJAY SAMUEL: I started a few years ago for two reasons. One, we were looking to introduce tracing inside the company, and at that time, OpenTracing and OpenCensus was converging into OpenTelemetry. We started evaluating OpenTelemetry for that. And given that we were moving into OpenTelemetry for tracing, I also went through the journey of migrating our metrics collection into OpenTelemetry. That’s basically how I got involved.

DANIEL GOMEZ BLANCO: How did I get involved in OpenTelemetry? I got involved through my work at Skyscanner, as an end user. I was driving adoption and open standards for telemetry. During COVID there was a need for simplification and how we approach infrastructure, how we approach, how we collect, how we process, and how we export telemetry data, and also basically… to basically lead the adoption of open standards and their simplification effort. So as an observability lead, I got more involved in the community aspect of OpenTelemetry, decided to interact with all their end users and meeting people that want to solve the same problems and want to find a solution that works for everyone.

DOUG ODEGAARD: So, OpenTelemetry, I actually, for several years, in my previous position, I was hired to actually develop observability software. I was writing my own thing, we were doing a lot of alert management and various things. It was so much work and I thought, this has got to be easier. Plus I wanted to make sure that it could be future, future proof, dare I use that term? But also extensible.

And when I discovered OpenTelemetry, I was just like, oh, thank you. Because it’s something that the company could carry forward. And also we didn’t have to worry about storing the data as much. And so it’s really provided a really excellent platform so that we can focus on the task at hand versus how to do the job. So how I got involved in the project was actually first as a customer. It was about three, close to four years ago, kind of the infancy of OpenTelemetry. And I would go online, I would look at the documentation, or I would be in the code a lot, but I wanted to learn more. So I would go to a SIG call and there would be someone from Google and Microsoft and other companies, and then there was this guy from this small fintech in the US. And at first it was a little awkward, but they were so excited to have me in the call because I was an end user. And so it really was, it was a wonderful experience to begin that way, to realize that I could contribute to this rather than simply be a consumer of it. So it was great. And then I transitioned my career into working for a vendor, and we implement these systems now for customers like myself that I was years ago. So it’s kind of a pay it forward, give back type of thing.

ADNAN RAHIĆ: How did we get involved into the OpenTelemetry project? We started contributing more to the blog with you guys started contributing a bit to the docs as well. And yeah, it’s just been a whole-hearted effort in the team to always kind of dedicate a few, a few minutes of each day to check out the OpenTelemetry project, find a way to contribute.

RYNN MANCUSO: I got involved in the OpenTelemetry project…honestly, I was working at one observability company in marketing, and they didn’t see the point. They didn’t want me to get involved. And I really believed in open source. I’d worked in Mozilla and Wikimedia and really believed that, like, this was the way forward from a strategic perspective. So the second I could switch to a company that did let me get involved, that’s what I did. And now I’m at Honeycomb. And I’m glad to say within the first three months, I made project member and started working with the End User Working Group and worked to grow it into a SIG, into all the programs that it has today, together with others.

5- What’s your favorite telemetry signal?

IRIS DYRMISHI: Tracing is my favorite signal.

SEVERIN NEUMANN: My favorite signal now is profiling, because I think this is really closing a big gap that was missing in observability, right? So I mentioned before, right, I come from the APM space, and now for me, APM, observability, it’s very hard to make, like, a difference here. But one thing that when I talk with people using APM products right now is they’re like, hey, where’s code level visibility with OpenTelemetry, right? My commercial agent is giving me that line of code that is breaking something. And this is what we get with profiling. And that’s why I’m really, really excited about it.

KAYLA REOPELLE: To decide a favorite signal is kind of difficult for me. I really love the power of traces. I think that traces can tell stories in ways that are very meaningful. But on the same, like, on the other hand, I’ve been so immersed in logs and trying to allow logs to have more connections to spans and traces, I definitely have a soft spot for logs as well.

MORGAN MCLEAN: I mean, I’m partial to distributed traces because that’s where this project got its start. And I think early on, that’s where a lot of the value was. No one else was really doing standardized distributed trace collection right? There were some open source examples of it attached to, like, Zipkin and Jaeger. But I think the reason OpenTelemetry got so much traction so quickly is that it was providing that.

I’m also partial to logs, which we launched last year, just because that’s one where, like, I’ve been involved in a lot of parts of OTel… But that’s one where like, I was involved in a lot of the core specification early on in driving that. And so it was really exciting to see that ship. Also, logs are just a thing that throughout my career before working on any of this, I just get frustrated with, because they’re never standardized, slow to process, they’re expensive. OTel going to bring a lot of changes there for the better for everyone who uses logs.

And finally, I guess profiles, because I work on that now. When I was at Google many years ago, I launched what I think was the world’s first distributed continuous profiling product, at least publicly available one, which was Google cloud profiling, Stackdriver profiling, they still support it, I still think it’s free, it’s very powerful. But profiling has always been a bit of a niche thing. Like, I know, like at Splunk and other companies, we support it, but it’s not as well known as metrics, and traces, and logs. I think with OTel, starting later this year, we’re gonna launch like full support for profiles. That’s really gonna change. Like, we had customers at Google who would spend an hour of our profiler and save like 20, 30% of their aggregate compute because they found some really poorly optimized code really quickly. For more people to have that ability and speed things up and for developers actually to get insight into how things work, that’s super exciting. Like, the tech has been there a long time and OTel bringing this mainstream is huge.

HENRIK REXED: When people ask me, who is your favorite kid? Usually I say, I don’t have a favorite kid, you know. All my kids are wonderful. They all have, I don’t know, a great thing, you know, out of it. So I think I love traces because sometimes it helps you to understand where it slowed down. I love metrics because as a performance engineer, I used to use metrics a lot. And I love logs because logs at the end, there’s no sampling. So if you just do analytics on logs, wow, you are so much precise.

So I don’t think I have a favorite signal. I’ll just say that depending on what I need and pick and choose, there’s clearly one signal that will help me more. There’s one thing that I’m very eager and waiting since Valencia is continuous profiling, because I love profiling and I think traces is great, but if there is a problem somewhere, profiling would be so much helpful. So I think, yeah, I don’t answer your questions, but I say, yeah, I love all the signals provided by OpenTelemetry.

VIJAY SAMUEL: I am thoroughly biased towards metrics. I feel metrics are the most powerful signal. As long as you are thinking through your instrumentation and making sure that you have the right granularity cardinality being sent in, to the platform, you can do powerful, powerful things with regards to anomaly detection, machine learning and many other things. So I love metrics.

DANIEL GOMEZ BLANCO: I mean, I have to say traces, because they give you the context. Traces give you the backbone correlation for all the other signals, right? But I do think that the current design of the API design of metrics is so powerful that I’m like falling in love again with metrics because of that way that we decouple instrumentation and measurement from aggregation of metrics is so powerful and so much richness to basically give us a way to describe our systems, that I’m falling back again in love with metrics.

DOUG ODEGAARD: My favorite signal, I have to say, I’m partial to traces because I’ve been doing software development for so long that that was the first thing that really turned me on to it was the ability to see that, especially because I know what it’s like, like to debug. But it’s also, I also know what it’s like in an incident to have to focus in very quickly. So yes, traces are my favorite, but I do also like to send that trace ID and span ID into the logs now. It’s kind of becoming my next favorite.

ADNAN RAHIĆ: My favorite signal is traces. I’m going to say traces, definitely. My favorite singer is Ed Sheeran.

RYNN MANCUSO: What is my favorite signal? I mean, I work for Honeycomb, so I am constitutionally obliged to say traces are my favorite signal.

Join us!

If you have a story to share about how you use OpenTelemetry at your organization, we’d love to hear from you! Ways to share:

Join the #otel-sig-end-user channel on the CNCF Community Slack
Join our OTel in Practice sessions
Share your stories on the OpenTelemetry blog
Contact us on the CNCF Community Slack for any other types of sessions you’d like to see!

Be sure to follow OpenTelemetry on Mastodon and LinkedIn, and share your stories using the #OpenTelemetry hashtag!

And don’t forget to subscribe to our YouTube channel for more great OpenTelemetry content!

Demystifying Observability 2.0

Adriana Villela — Mon, 10 Jun 2024 19:21:13 +0000

Our systems have gotten complex. Like really complex. Organizations have mostly shifted from monoliths to microservices. They’ve embraced the Cloud, and with it, Kubernetes (PS: happy 10th b-day to Kubernetes!) and all sorts of other cloud native tools that help run the things that we’ve grown accustomed to having in our tech-dependent lives: access to government services, social media, airline booking, shopping, streaming services, and so on.

As our systems get more and more complex, engineers need a way to understand them when things go 💩, so that services can be restored in a timely manner.

Enter Observability, which helps with just that. Observability has been around for a while now, and it’s been really exciting to see so many organizations embarking on their respective observability journeys.

Now, if you’ve been following the interwebs, you may have heard some rumblings about Observability 2.0. Cool. But what is it really, and how does it differ from Observability 1.0? Well, you’ve come to the right place. Sit back, relax, and let me take you on a journey.

Defining Observability

Before we get into Observability 1.0 vs 2.0, let’s start with a definition of Observability, also known as o11y to us folks who sometimes get lazy and don’t want to write out the whole word. 🙃 (For the uninitiated: o11y == the 11 letters between “o” and “y” in “Observability”.)

The “classic” definition of Observability comes from control theory:

Observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs.

– Rudolf E. Kálmán

This definition was popularized by Charity Majors.

I love this definition, and I’ve used it for many years, including in my very first blog post on Observability, and more recently, in my O’Reilly Observability video course.

That being said, there’s a refinement to the definition of Observability that I’ve been embracing of late, which was coined by my good friend, Hazel Weakly, who has an amazing blog post on redefining Observability. (Hazel is also incredibly smart and super astute and you should totally follow her on LinkedIn):

Observability is the process through which one develops the ability to ask meaningful questions, get useful answers, and act effectively on what you learn.

– Hazel Weakly

It’s so simple, and so elegant, and I love it. Also, it applies to both Observability 1.0 and 2.0, and does not hold us back from continuing to refine Observability.

Okay, now that we’ve gotten the basics out of the way, let’s tackle this 1.0 vs. 2.0 business.

I set out to write this piece because I’ve found myself talking a lot about Observability 2.0 recently, including last week on Whitney Lee’s Enlightning show, and in an upcoming episode of The Cloud Gambit. After all this talking about it, I wanted a place to jot down my thoughts, and to also share them with y’all. I honestly thought it would be a straight regurgitation of what I’d already said. But then I asked Hazel to look over this piece, and her feedback encouraged me to think about this further, thereby refining some of my understanding and thoughts around this. Which is awesome, because it’s so fitting, given that I’m talking about the evolution of our understanding of Observability!

Observability 1.0

When Observability burst onto the scene, it was still a very APM-dominated world. Many APM vendors, sensing that Observability was becoming an Actual Thing, pivoted to Observability. This pivot, however, was mostly in name only, in much the same way that many organizations pivoted from Ops to DevOps (or SRE or Platform Engineering) in name only. New name, but business as usual. And perhaps we can’t blame them for that. These are paradigm shifts and paradigm shifts are often hard to swallow. You’ve gotta start somewhere, and maybe a name change is as good a place as any.

So, time for the big reveal…Observability 1.0 is APM. But more specifically, what is Observability 1.0? Observability 1.0 is focused on:

1- Yow you operate your code

This means that it’s more of an Ops concern, and not so much of an Everyone concern.

2- Known unknowns

Also known as “predicable shit happens”. We know the usual things that go wrong with our systems, and we put dashboards in place to represent all of things that we know can go wrong with our systems (and for which we know the fixes), so that we can keep an eye on things if they go sideways.

3- Multiple sources of truth

These “sources of truth” are traces, metrics, and logs, also known as “The Three Pillars”. I actually hate that term, because it implies that these things are siloed from one another (more on that later). I much prefer the term “signal”. A signal is anything that gives you data.

I suppose that the whole Three Pillars thing kind of makes sense for Observability 1.0, where traces, metrics, and logs were often not correlated. This is especially true since, in the early days of Observability, we didn’t really have a common language for even talking about these signals. Each vendor had their own standard, and that may or may not have included a way to correlate the three signals.

I also want to add that there was much more of an emphasis on logs and metrics, because that’s just something that developers and operators are familiar with. Traces have been around, but were not very widely used.

Observability 2.0

So now that we know what Observability 1.0 is all about, let’s look at how it differs from Observability 2.0.

First things first. Credit where credit is due. The term “Observability 2.0” was coined by Charity Majors. Observability 2.0 is the acknowledgment that Observability, like all things tech and non-tech, continues to evolve. The evolution to Observability 2.0 is the recognition that we made a decent stab at Observability (i.e. 1.0), but unfortunately, it didn’t really fulfill the promise of the definition of Observability that we saw earlier on. No problem, because things are constantly evolving.

So what makes Observability 2.0 different from 1.0? It has the following characteristics:

1- It’s focused not only on how you operate your code, but also on how you develop your code

This means that Observability is part of the systems development lifecycle (SDLC), and is therefore a concern of developers, QAs, and SREs. How?

Developers instrument their code so that they can troubleshoot it during development. 🤯 Instrumentation is the process of adding code to software to generate telemetry signals for Observability purposes. Software engineers already rely on logging for troubleshooting (hello, “print” statements?), so why not add traces and metrics into the mix?

Quality Assurance (QA) analysts leverage instrumented code during testing. When they encounter a bug, QAs can use telemetry data to enable them to troubleshoot code and file more detailed bug reports to developers. Or, if they’re unable to troubleshoot the code with the telemetry provided, it means that the system has not been sufficiently instrumented. Again, they go back to developers with that information so that developers can add more instrumentation to the code.

QAs further take advantage of instrumented code by creating trace-based tests (TBT) for integration testing. In a nutshell, TBT leverages traces to create integration tests. For anyone interested in seeing TBT in action, the OpenTelemetry Demo leverages TBT using the opens source version of Tracetest.

SREs leverage instrumented code to create service-level objectives (SLOs). SLOs help us answer the question, “What is the reliability goal of this service?” SLO are based on Service Level Indicators (SLIs), which are themselves based on metrics. Metrics that were instrumented by your developer! 🤯 SREs can create alerts based on these SLOs, so that when an SLO is breached, they’re notified right away. Furthermore, since the SLO is ultimatley tied a metric (via an SLI), which was correlated to a trace (more on signal correlation shortly), the SRE knows where to start looking when an issue arises in production.

**CI/CD pipelines are instrumented. **CI/CD pipelines are the backbone of modern SDLC. They are responsible for packaging and delivering code to production in a timely manner. When they fail, we can’t get code into production, which means angry users. Nobody likes angry useres. Ever. Therefore, having observable CI/CD pipelines allows us to address pipeline failures in a more timely manner to help alleviate software delivery bottlenecks.

2- It’s focused on unknown unknowns

Also known as “unpredictable shit happens”. Let’s face it, you can’t know every problem that there’s ever going to be. This is especially true in the world of microservices, where services interact with each other in such weird and unpredictable ways because…well, we users tend to use systems in very weird and unpredictable ways! 🤯 Traditional dashboards can’t save you, but SLO-based alerts can.

3- It’s focused on a single source of truth: events

Wait…what? What about traces, metrics, and logs? Well, traces, metrics, and logs all types of events. An event is information about a thing that happened. They are structured (think JSON-like), and timestamped. Traces, metrics, and logs are therefore different types of events that serve different and important purposes, each contributing to the Observability story. Furthermore, they’re all correlated. Instead of Three Pillars, they’re more like the three strands that make up a braid (shoutout to my teammate Ted Young for this analogy).

In addition, we now have a common standard for defining and correlating traces, metrics, and logs: OpenTelemetry. Most Observability vendors are all in on OpenTelemetry, which means that it has become the de-facto standard for instrumenting code (and also the second most popular CNCF project in terms of contributions 🎉). It also means that these vendors all ingest the same data, and it’s up to how those vendors render the data that differentiates them from one other.

I also want to add that in this Observability story, we place traces front and center, since they help give us that end-to-end picture of what happens when someone does a thing to a system, with metrics and logs serving as supporting actors which add useful details to that picture. And of course, everything correlated.

Final thoughts

Observability has come a long way from its early days, and Observability 2.0 is the acknowledgement that Observability is evolving, and most importantly, that we’re getting closer and closer to fulfilling the promise of Observability itself.

I can’t wait to see what the future has in store!

Now, please enjoy this photo of my rat Katie, enjoying some hangtime in the pocket of my husband’s bathrobe. 💜

Until next time, peace, love, and code. ✌️💜👩‍💻

Things You Might Not Have Known About the OpenTelemetry Operator

Adriana Villela — Thu, 25 Apr 2024 04:00:00 +0000

The OpenTelemetry (OTel) Operator is a Kubernetes Operator that manages OTel for you in your Kubernetes cluster to make life a little easier. It does the following:

Manages deployment of the OpenTelemetry Collector, supported by the OpenTelemetryCollector custom resource (CR).
Manages the configuration of a fleet of OpenTelemetry Collectors via OpAMP integration, supported by the OpAMPBridge custom resource.
Provides integration with the Prometheus Operator's PodMonitor and ServiceMonitor CRs.
Injects and configures auto-instrumentation into your pods, supported by the Instrumentation custom resource.

I first used the Operator last year when I used Kratix to demonstrate how a platform team can make the OTel Operator available to developers, so that they can self-provision a pre-configured OTel Collector in their Kubernetes clusters. More recently, I’ve had a chance to get to know the Operator a bit better, especially while putting together my O’Reilly video course on Observability with OpenTelemetry, and in the recent KubeCon EU talk that I did with Reese Lee.

I’ve learned some really cool things, and I thought it might be helpful to share some little OTel Operator goodies that I’ve picked up along the way, in the form of a Q&A.

Please note that this post assumes that you have some familiarity with OpenTelemetry, the OpenTelemetry Collector, the OpenTelemetry Operator (including the Target Allocator), and Kubernetes.

Q&A

Q1: Does the Operator support multiple Collector configuration sources?

Short answer: no.

Longer answer: OTel Collector can be fed more than one Collector config YAML file. That way, you can keep your base configurations in, say, otelcol-config.yaml, and overrides or additional configurations of the base can go in, for example, otelcol-config-extras.yaml. See an example of this in the OTel Demo’s Docker compose file.

Unfortunately, while the OTel Collector supports multiple Collector configuration files, the Collector managed by the OTel Operator does not.

To get around this, you could merge the multiple Collector configs through some external tool beforehand. For example, if you were deploying the Operator via Helm, you could technically pass it multiple Collector config files using multiple --values flags and let Helm do the merging for you.

For reference, check out this thread in the #otel-operator CNCF Slack channel.

Q2: How can I securely reference access tokens in the Collector’s configuration?

In order to send OpenTelemetry data to an Observability backend, you must define at least one exporter. Whether you use OTLP or some proprietary vendor format, most exporters typically require that you specify an endpoint and an access token when sending data to a vendor backend.

When using the OpenTelemetry Operator to manage the OTel Collector, the OTel Collector config YAML is defined in the OpenTelemetryCollector CR. This file should be version-controlled and therefore shouldn’t contain any sensitive data, including access tokens stored as plain text.

Fortunately, the OpenTelemetryCollector CR gives us a way to reference that value as a secret. Here’s how you do it:

1- Create a Kubernetes secret for your access token. Remember to base-64 encode the secret.

2- Expose the secret as an environment variable by adding it to the OpenTelemetryCollector CR’s env section. For example:

 env:
    - name: LS_TOKEN
      valueFrom:
        secretKeyRef:
          key: LS_TOKEN
          name: otel-collector-secret

3- Reference the environment variable in your exporter definition:

exporters:
   otlp/ls:
     endpoint: "ingest.lightstep.com:443"
     headers:
       "lightstep-access-token": "${LS_TOKEN}"

For more info, check out my full example here, along with full instructions here.

Q3: Is the Operator version at parity with the Collector version?

For every Collector release, there is an Operator release which provides support for that Collector version. For example, at the time of this writing, the latest Operator version is 0.98.0. Thus, the the default image of the Collector used by the Operator is version 0.98.0 of the core distribution (as opposed to the contrib distribution).

Q4: Can I override the base OTel Collector image?

Yes!

As we saw earlier, the core distribution is the default Collector distribution used by the OpenTelemetryCollector CR. The Core distribution is a bare-bones distribution of the Collector for OTel developers to develop and test. It contains a base set of components–i.e. extensions, connectors, receivers, processors, and exporters.

If you want access to more components than the ones offered by core, you can use the Collector's Kubernetes Distribution instead. This distribution is made specifically to be used in a Kubernetes cluster to monitor Kubernetes and services running in Kubernetes. It contains a subset of components from
OpenTelemetry Collector Core and OpenTelemetry Collector Contrib.

If you want to use specific Collector components, you can build your own distribution using the OpenTelemetry Collector Builder (OCB), and include only the components that you need.

Either way, the OpenTelemetryCollector CR allows you to override the default Collector image with one that better suits your needs by adding spec.image to your OpenTelemetryCollector YAML. In addition, you can also specify the number of Collector replicas you wish to spin up, by adding spec.replicas (this is totally independent of whether or not you override the Collector image). Your code would look something like this:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otelcol
  namespace: mynamespace
spec:
  mode: statefulset
  image: <my_collector_image>
  replicas: <number_of_replicas>
...

Where:

<my_collector_image> is the name of a valid Collector image from a container repository
<number_of_replicas> is the number of pod instances for the underlying OpenTelemetry Collector

Keep in mind that if you're pulling a Collector image from a private container registry, you'll need to use imagePullSecrets. Since private container registries require authentication, this will enable you to authenticate against that private registry. For more info on how to use imagePullSecrets for your Collector image, check out the instructions here.

For more info, check out the OpenTelemetryCollector CR API docs.

Q5: Does the Target Allocator work for all deployment types?

No. The Target Allocator only works for StatefulSet, and DaemonSet (newly-introduced). More info here.

Q6: If I’m using Operator’s Target Allocator for Prometheus service discovery, do I need `PodMonitor` and `ServiceMonitor` CRs installed in my Kubernetes cluster?

Yes, you do. These CRs are bundled with the Prometheus Operator; however, they can be installed standalone, which means that you don’t need to install the Prometheus Operator just to use these two CRs with the Target Allocator.

The easiest way to install the PodMonitor and ServiceMonitor CRs is to grab a copy of the individual PodMonitor YAML and ServiceMonitor YAML custom resource definitions (CRDs), like this:

kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.71.2/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml

kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.71.2/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml

Check out my example of the OpenTelemetry Operator’s Target Allocator with ServiceMonitor here.

Q7: Do I need to create a service account to use the Target Allocator?

No, but you do need to do a bit of extra work. So, here’s the deal…although you need a service account to use the Target Allocator, you don’t have to create your own.

If you enable the Target Allocator and don’t create a service account, one is automagically created for you. This service account’s default name is a concatenation of the Collector name (metadata.name in the OpenTelemetryCollector CR) and -collector. For example, if your Collector is called mycollector, then your service account would be called mycollector-collector.

By default, this service account has no defined policy, you’ll still need to create your own ClusterRole and ClusterRoleBinding, and associate the ClusterRole to the ServiceAccount via ClusterRoleBinding.

See the Target Allocator readme for more on Target Allocator RBAC configuration.

Q8: Can I override the Target Allocator base image?

Just like you can override the Collector base image (as we saw in Q4), you can also override the Target Allocator’s base image.

Please keep in mind that it’s usually best to keep the Target Allocator and OTel operator versions the same, to avoid any compatibility issues. If you do choose to override the Target Allocator’s base image, you can do so by adding spec.targetAllocator.image in the OpenTelemetryCollector CR. You can also specify the number of replicas by adding spec.targetAllocator.replicas (this is totally independent of whether or not you override the TA image). Your code would look something like this:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otelcol
  namespace: mynamespace
spec:
  mode: statefulset
  targetAllocator:
    image: <ta_image_name>
    replicas: <number_of_replicas>
...

Where:

<ta_image_name> is a valid Target Allocator image from a container repository.
<number_of_replicas> is the number of pod instances for the underlying Target Allocator

Q9: If it’s not recommended that you override the Target Allocator base image, then why would you want to?

One use case might be if you need to host a mirror of the Target Allocator image in your own private container registry for security purposes.

If you do need to reference a Target Allocator image from a private registry, you’ll need to use imagePullSecrets. To use imagePullSecrets with the OTel Operator, check out the instructions here. Note that you don’t need to create a serviceAccount for the Target Allocator, since once is already created for you automagically if you don’t create one yourself (see Q6).

For more info, check out the Target Allocator API docs.

Q10: Is there a version lag between the OTel Operator auto-instrumentation and auto-instrumentation of supported languages?

If there is a lag, it's minimal, as maintainers try to keep these up to date for each release cycle. Keep in mind that there are breaking changes in some semconvs and the team is trying to avoid breaking users' code. More info here.

Final Thoughts

Hopefully this has helped to demystify the OTel Operator a bit more. There’s definitely a lot going on, and the OTel Operator can certainly be a bit scary at first, but understanding some of the basics will get you well on your way to mastering this powerful tool.

If you have any questions about the OTel Operator, I highly recommend that you post questions on the #otel-operator channel on the CNCF Slack. Maintainers and contributors are super friendly, and have always been more than willing to answer my questions! You can also hit me up, and I'll try my best to answer your questions, or to direct you to folks who have the answers!

And now I'll leave you with a photo of my rat, Katie Jr.

Until next time, peace, love, and code. ☮️❤️👩‍💻

OpenTelemetry Collector Anti-Patterns

Adriana Villela — Mon, 26 Feb 2024 20:17:34 +0000

The OpenTelemetry Collector is one of my favorite OpenTelemetry (OTel) components. It’s a flexible and powerful data pipeline which allows you to ingest OTel data from one or more sources, transform it (including batching, filtering, and masking), and export it to one or more observability backends for analysis. It’s vendor-neutral. It’s extensible, meaning that you can create your own custom components for it. What’s there not to like?

Unfortunately, as it happens with many tools out there, it is also very easy to fall into some bad habits. Today, I will dig into five OpenTelemetry Collector antipatterns, and how to avoid them. Let’s get started!

Antipatterns

1- Improper use of Collector deployment modes

It’s not just enough to use a Collector. It’s also about how your Collectors are deployed within your organization. That’s right - Collector*s*, plural. Because one is often not enough.

There are two deployment modes for Collectors: agent mode and gateway mode, and both are needed.

In agent mode, the Collector sits next to the application or on the same host as the application.

In gateway mode, telemetry data is sent to a load balancer, which then determines how to distribute the load amongst a pool of Collectors. Because you have a pool of Collectors, should one Collector in that pool fail, one of the other Collectors in the pool can take over. This keeps data flowing to your destination sans disruptions. Gateway mode is commonly deployed per cluster, data center, or region.

So which should you use? Both agent and gateway.

If you’re collecting telemetry data for your application, place a Collector agent alongside your application. If you’re collecting data for infrastructure, place a Collector agent alongside your infrastructure. Whatever you do, don’t collect telemetry for all of your infrastructure and applications using a single Collector. That way, if one Collector fails, the rest of your telemetry collection is unaffected.

The telemetry from your Collector agents can then be sent to a Collector gateway. Because the gateway sits behind a load balancer, you don’t have a single point of failure for exporting telemetry data, typically to your observability backend.

Bottom line: Having the right Collector deployment configuration to send data to your observability backend ensures higher availability of your telemetry collection infrastructure.

2- Not monitoring your Collectors

Deploying multiple Collector agents and a Collector gateway is great, but it’s not good enough. Wouldn’t it be nice to know when one of your Collectors is malfunctioning, or when data is being dropped? That way, you can take action before things start to escalate. This is where monitoring your Collectors can be very useful.

But how does one monitor a Collector? The OTel Collector already emits metrics for the purposes of its own monitoring. These can then be sent to your Observability backend for monitoring.

3- Not using the right Collector Distribution (or not building your own distribution)

There are two official distributions of the OpenTelemetry Collector: Core, and Contrib.

The Core distribution is a bare-bones distribution of the Collector for OTel developers to develop and test. It contains a base set of extensions, connectors, receivers, processors, and
exporters.

The Contrib distribution is for non-OTel developers to experiment and learn. It also extends the Core distribution, and includes components created by third-parties (including vendors and individual community members), that are useful to the OpenTelemetry community at large.

Neither Core nor Contrib alone are meant to be part of your production workload. Using just Core by itself is too bare-bones and wouldn’t suit an organization’s needs. (Though its components are absolutely needed!) And although many OpenTelemetry practitioners, deploy Contrib in their respective organizations, it has many components, and you likely won’t need every single exporter, receiver, processor, connector, and extension. That would be overkill, and your Collector instance ends up needlessly bloated, potentially increasing the attack surface.

But how do you pick and choose the components that you need? The answer is to build your own distribution, and you can do that using a tool called the OpenTelemetry Collector Builder (OCB). In addition, at some point, you may need to create your own custom Collector component, such as a processor or exporter. The OCB allows you to integrate your custom components AND pick and choose the Contrib components that you need.

It is also worth mentioning that some vendors build their own
Collector distributions. These are OTel Collector distributions that are curated to Collector components that are specific to that vendor. They may be a combination of custom, vendor-developed components, and curated Collector Contrib components. Using vendor-specific distributions ensures that you are using just the Collector components that you need, again reducing overall bloat.

Bottom line: Using the right distribution reduces bloat and allows you to include only the Collector components that you need.

4- Not updating your Collectors

This one’s short and sweet. Keeping software up-to-date is important, and the Collector is no different! By regularly updating the Collector, it allows you to stay up-to-date with the latest version so that you can take advantage of new features, bug fixes, performance improvements, and security fixes.

5- Not using the OpenTelemetry Collector where appropriate

OpenTelemetry allows you to send telemetry signals from your application to an observability backend in one of two ways:

Sending telemetry data “direct from application” for non-production systems is all well and good if you’re getting started with OpenTelemetry, but it is neither suited nor recommended to use this approach for production systems. Instead, the OpenTelemetry docs recommend using the OpenTelemetry Collector. How come?

Per the OTel Docs, the Collector “allows your service to offload data quickly and the collector can take care of additional handling like retries, batching, encryption or even sensitive data filtering.”

Check out some additional Collector benefits:

Collectors can enhance the quality of the telemetry emitted by an application while also minimizing costs. For example: sampling spans to reduce costs, enriching telemetry with extra metadata, and generating new telemetry, such as metrics derived from spans.
Using a Collector to ingest telemetry data makes it easy to change to a new backend or export the data in a different format. If we want to change how telemetry is being processed or exported, that change happens in one place (the Collector!), as opposed to making the same change for multiple applications in your organization.
Collectors allow you to receive data of various formats and translate to the desired format for export. This can be very handy when transitioning from some other telemetry solution to OTel.
Collectors allow you to ingest non-application telemetry. This includes logs and non-app metrics from infrastructure like Azure, Prometheus, and Cloudwatch.

That being said, there are some use-cases where folks don't want or can't use a Collector. For instance, when collecting data at the edge from IOT devices, it might be better to send data directly to their observability backend instead of a local Collector, given that resources on that edge might be limited.

Bottom line: As a general rule, using the OpenTelemetry Collector gives you additional flexibility for managing your telemetry data.

Final Thoughts

The OpenTelemetry Collector is a powerful and flexible tool for ingesting, manipulating, and exporting OpenTelemetry data. By using it to its full potential and by avoiding these five pitfalls, your organization can be well on its way towards achieving observability greatness.

Now, please enjoy this lovely photo of my rats Buffy, Mookie, and Katie.

Until next time, peace, love, and code! ☮️❤️👩‍💻

Tracetest in Action: Running Trace-Based Tests on the OpenTelemetry Demo App with Nomad

Adriana Villela — Tue, 28 Feb 2023 15:00:00 +0000

The concept of Trace-based testing (TBT) was first presented by my awesome colleague, Ted Young, at KubeCon North America 2018. The idea behind it was simple, yet elegant: if you’re already creating distributed Traces in your application code, why not use this same Trace data to create test assertions to validate your end-to-end system flow?

Back in 2018, Trace-based testing was just an idea. Fast-forward to today: TBT is now a reality, thanks to Trace standardization à la OpenTelemetry (OTel) and Trace-based testing tools like Tracetest, Helios, and Malabi.

I first heard about TBT in the summer of 2022, when I was writing a piece on how Observability is useful for QAs. After writing the piece, I wanted to get my hands dirty and try this Trace-based testing thing for myself. The tool that I chose for my initial TBT explorations was Tracetest, which is open-source.

Tracetest Overview

Tracetest came out in May 2022, and I first got my hands on Tracetest back in June 2022, so it was still pretty fresh! At the time, it integrated only with Jaeger. More specifically, it used the Jaeger API to pull OTel Traces from Jaeger to register them into Tracetest. You then used the UI to create trace-based tests.

Tracetest has come a long way since those early days! It now has:

A CLI
A quickstart installer so that you can get a full-fledged example up and running on an existing Kubernetes cluster (Very slick and easy to get going, I might add!)
Integrations with a number of Observability back-ends and other tools, including Lightstep.
A declarative approach to defining Trace-based tests via YAML

I got to play around with these newer features last December, after a months-long hiatus, and it was really cool to see the evolution of the product. If you follow my work, you’ll know that I play in both the Kubernetes and Nomad worlds. Today, I’ll be taking you on a quick little guided tour of Tracetest, using Traces from the OpenTelemetry Demo App to give you a feel for how it works. The whole setup will be running on HashiCorp Nomad. \

Tutorial

In this tutorial, I will show you how to install Tracetest (v0.9.1) and the OTel Demo App (v1.1.0) on Nomad running locally using HashiQube. I will then show you how to create and run a simple test with Tracetest, using a couple of Traces from the OTel Demo App.

Although the versions of both Tracetest and the OTel Demo App used in this tutorial are slightly older than what’s out there, you should still get a decent idea of how everything works. The OTel Demo App is configured to send Traces and Metrics to Lightstep via the OTel Collector. The Collector is also configured to send Traces to Tracetest. This configuration is based on guidance from the official Tracetest docs. You can see the full Collector configuration used in this example here.

Please note that there are no official Nomad jobspecs for either Tracetest or the OTel Demo App, so I went ahead and did the conversions myself from Kubernetes manifests to Nomad jobspecs. You can check out the jobspecs in this repo. If you’re curious as to how I went about the Kubernetes-to-Nomad conversion, you can check out my blog post on this topic. I also have a blog post dedicated to running the OTel Demo App on Nomad. If this tickles your fancy, you can check it out here.

Assumptions

Before we move on, I am assuming that you have a basic understanding of:

Nomad. If not, mosey on over to my Nomad intro post. This blog post by Daniela Baron is also great.
Observability (o11y) and OpenTelemetry (OTel). If not, mosey on over to my Observability & OTel post.

Pre-Requisites

In order to run the example in this tutorial, you’ll need the following:

Vagrant (version 2.3.1 at the time of this writing) - to provision HashiQube
Docker (version 20.10.21 at the time of this writing) - we’ll be running HashiQube on Docker, using the Vagrant Docker Provider.

You’ll also need to make sure that your Docker Desktop resource settings are set to the following:

CPUs: 3
Memory: 9.5GB
Swap: 3GB

Tutorial Repos

Below are the repos that we’ll be using for today’s tutorial:

My modified HashiQube Repo (fork of servian/hashiqube). If you’re curious, you can see what modifications I’ve made here.
My Nomad Conversions repo

Part 1: Provision Nomad, Consul, and Vault using HashiQube

In this section, we’ll be provisioning a local Hashi environment (including Nomad, Consul, and Vault) instance using HashiQube. Then, we’ll install Tracetest and the OTel Demo App on Nomad. Feel free to skip this section if you already have a working Hashi environment with Nomad, Consul, and Vault.

1- Update /etc/hosts

We use the Traefik load-balancer to expose our services, which we access as subdomains of localhost. In order ensure that we can access our Traefik-exposed services (and also the Traefik dashboard itself, you’ll need to add the following entries to /etc/hosts on your host machine:

127.0.0.1 traefik.localhost
127.0.0.1 otel-demo.localhost
127.0.0.1 tracetest.localhost

For more on Traefik load-balancing on Nomad, check out this post.

2- Provision a Local Hashi Environment with HashiQube

Start HashiQube by following the detailed instructions here.

You can now access the apps below:

Part 2: Deploying Tracetest and the OTel Demo App on Nomad

We’re ready to deploy the OTel Demo App and Tracetest on Nomad!

1- Add Lightstep Access Token to Vault

As I mentioned earlier, we’ll be sending our Traces to both Lightstep and Tracetest. In order to send Traces to Lightstep, you’ll need to:

Get a Lightstep Access Token. (Make sure that you sign up for a Lightstep account first, if you don’t already have one.)
Configure Vault by following the instructions here.
Add your Lightstep Access Token to Vault by running the command:

vault kv put kv/otel/o11y/lightstep ls_token="<LS_TOKEN>"

Where <LS_TOKEN> is your Lightstep Access Token.

2- Deploy the OTel Demo App and Tracetest

First, let’s clone the repo, and go to our working directory:

git clone https://github.com/avillela/nomad-conversions.git
cd nomad-conversions

Next, let’s deploy the services:

# Enable memory over-subscription
nomad operator scheduler set-config -memory-oversubscription true
# Tracetest deploy
nomad job run -detach tracetest/jobspec/traefik.nomad
nomad job run -detach tracetest/jobspec/postgres.nomad
nomad job run -detach tracetest/jobspec/tracetest.nomad
nomad job run -detach tracetest/jobspec/otel-collector.nomad
nomad job run -detach tracetest/jobspec/go-server.nomad
# OTel Demo App deploy
nomad job run -detach otel-demo-app/jobspec/redis.nomad
nomad job run -detach otel-demo-app/jobspec/ffspostgres.nomad

nomad job run -detach otel-demo-app/jobspec/adservice.nomad
nomad job run -detach otel-demo-app/jobspec/cartservice.nomad
nomad job run -detach otel-demo-app/jobspec/currencyservice.nomad
nomad job run -detach otel-demo-app/jobspec/emailservice.nomad
nomad job run -detach otel-demo-app/jobspec/featureflagservice.nomad
nomad job run -detach otel-demo-app/jobspec/paymentservice.nomad
nomad job run -detach otel-demo-app/jobspec/productcatalogservice.nomad
nomad job run -detach otel-demo-app/jobspec/quoteservice.nomad
nomad job run -detach otel-demo-app/jobspec/shippingservice.nomad
nomad job run -detach otel-demo-app/jobspec/checkoutservice.nomad
nomad job run -detach otel-demo-app/jobspec/recommendationservice.nomad
nomad job run -detach otel-demo-app/jobspec/frontend.nomad
nomad job run -detach otel-demo-app/jobspec/loadgenerator.nomad
nomad job run -detach otel-demo-app/jobspec/frontendproxy.nomad
nomad job run -detach otel-demo-app/jobspec/grafana.nomad
nomad job run -detach otel-demo-app/jobspec/jaeger.nomad
nomad job run -detach otel-demo-app/jobspec/prometheus.nomad

NOTE: We’ve enabled memory oversubscription in Nomad. This is a one-time setting.

Since we’re running the jobs in detached mode, Nomad won’t wait to start the next job until the current one has deployed successfully. This means that your output will look something like this:

Job registration successful
Evaluation ID: d3eaa396-954e-241f-148d-6720c35f34bf
Job registration successful
Evaluation ID: 6bba875d-f415-36b7-bfeb-2ca4b9982acb
Job registration successful
Evaluation ID: 16dc8ef8-5e26-68f4-89b6-3d96b348775b
Job registration successful
Evaluation ID: 34de0532-a3b5-8691-bf18-51c0cc030573
Job registration successful
Evaluation ID: 7310e6a2-9945-710b-1505-c01bd58ccd35
...

Friendly reminder that the Evaluation ID values will be different on your machine.

3- See it in Nomad

As things are deploying, you can mosey on over to the Nomad UI at http://localhost:4646 to see how things are coming along:

It will take some time for all of the services to come up (sometimes up to 30 minutes), depending on network speed (especially since Nomad needs to download the images and initialize the services) and system resources, so be patient! Since some services depend on other services in order to run, you may see services in limbo or some going up and down for a while, per the above screen capture. DON’T PANIC! IT WILL ALL BE OKAY!!

Once all of the jobs are up and running, you’ll see everything look green, like this:

You can also head on over to Consul at http://localhost:8500 to see the health of the services:

By default, unhealthy services show up at the top, with a red “x” next to them. Since we don’t see any nasty red “x”s in the above screen shot, we know that our services are lookin’ good!

4- Access the OTel Demo App

To make sure that the OTel Demo App is up and running, you can access it at: http://otel-demo.localhost.

5- Access Tracetest

You should also be able to access the Tracetest UI at: http://tracetest.localhost

Part 3: Creating & Running Tests in Tracetest

Now that we have the OTel Demo App and Tracetest up and running, we can finally create our tests in Tracetest!

1- Install the Tracetest CLI

The Tracetest CLI allows you to interact with Tracetest programmatically. You can install the Tracetest CLI on Mac via Homebrew:

brew install kubeshop/tracetest/tracetest

For Windows and Linux, check out the instructions here.

Let’s make sure that the CLI is installed properly:

tracetest version

Sample output:

v0.9.1

2- Configure the Tracetest CLI

Now that the CLI is installed, let’s configure the Tracetest CLI so that it knows what Tracetest installation it needs to talk to. Make sure that you’re still in the nomad-conversions root directory, and then run:

cd tracetest
tracetest configure

This will create a file called config.yaml in the directory from which you ran tracetest configure.

When you run the above command, you will be prompted for the Tracetest server URL:

Enter your Tracetest server URL [http://tracetest.localhost]:

Notice that it’s already pre-populated with http://tracetest.localhost. This is because there’s already a config.yaml file in our tracetest directory. If you don’t type anything in and hit return, the CLI will just use the value already in config.yaml. If you want to use a different URL, make sure that you prefix that URL with http:// (or https:// if you’re hosting a secure instance).

Next, you’ll be prompted on whether or not you wish to enable analytics:

Enable analytics? [Y/n]:

If you hit return here, it will enable analytics by default. Type n to disable analytics.

Now, let’s open up config.yaml:

cat config.yaml

Sample output:

scheme: http
endpoint: tracetest.localhost
analyticsEnabled: false

3- Create a Trace-based test in Tracetest

Now we’re ready to create a test! We can do this in one of two ways:

Using the UI (check out the Tracetest docs here for details)
From a YAML file

I’m a big fan of doing things programmatically, so let’s create our test from a YAML file. Per the Tracetest docs, you can create tests from HTTP requests, gRPC requests, cURL commands, and more. In our case, we’re creating our tests from a gRPC request, so our Tracetest test definition file YAML looks like this:

type: Test
spec:
 id: <test_id>
 name: <test_name>
 description: <test_description>
 trigger:
   type: grpc
   grpc:
     protobufFile: |
        <protobuf_file_contents_here>
     address: <grpc_endpoint_address>
     method: <grpc_method_name>
 specs:
 - selector: span[tracetest.span.type="general" name="<operation_name>"]
   assertions:
   - attr:<attribute_name> <operator> <value>

Where:

<test_id> is a unique test identifier. If you create your test via the UI, a unique ID is assigned. If you create a test from a YAML file, you can assign your own ID. For this example, let’s name our ID tt-recsvc-01. As long as it’s unique, you should be good.
<test_name> is the name of your test. Use something short and meaningful.
<test_description> is short description of your test.
trigger.type is grpc, since we’re creating a test from a gRPC request.
trigger.grpc.protobufFile should be the complete contents of your protobuf file. In our case, we’re going to embed the contents of OTel Demo App’s demo.proto file in this field.
<grpc_endpoint_address> is the address at which the gRPC endpoint is served. In this case, we’re using Consul DNS to expose the service’s address. If we look at the Recommendation Service’s Nomad jobspec, you’ll see that the name of the gRPC service is recommendationservice. So when we query it in Consul, it should be accessible at this address recommendationservice.service.consul. We can test this by logging into the HashiQube image. Do this by going to the root directory of HashiQube repo, and typing vagrant ssh. Once you’re in the HashiQube Vagrant box, run this command:

dig +short recommendationservice.service.consul

Expected result:

172.17.0.2

We get the port number from the Recommendation Service jobspec’s port definition stanza. In our case, the port number is 9001. Note that we need to use a static port number in order for this setup to work.

<grpc_method_name> is the name of the gRPC method that we’re testing. It follows the naming convention <package_name>.<service_name>.<method_name>. Where <package_name> is hipstershop, <service_name> is RecommendationService, and <method_name> is ListRecommendations.
spec is where we define our test assertions. We need to first select our target Span(s), and then we define what assertions to apply to the Span(s). In our case, we are selecting a span named get_product_list, and we’re asserting that the value of the attribute app.filtered_products.count is set to 9.

With all that in mind, our YAML should now look like this:

type: Test
spec:
id: tt-recsvc-01
name: RecommendationService test
description: Sample test file for the OTel Demo App RecommendationService
trigger:
  type: grpc
  grpc:
    protobufFile: |
      // Copyright 2020 Google LLC
      //
      // Licensed under the Apache License, Version 2.0 (the "License");
      // you may not use this file except in compliance with the License.
      // You may obtain a copy of the License at
      //
      //      http://www.apache.org/licenses/LICENSE-2.0
      //
      // Unless required by applicable law or agreed to in writing, software
      // distributed under the License is distributed on an "AS IS" BASIS,
      // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
      // See the License for the specific language governing permissions and
      // limitations under the License.

      syntax = "proto3";

      import "google/protobuf/timestamp.proto";

      package hipstershop;

      option go_package = "genproto/hipstershop";

      // -----------------Cart service-----------------

      service CartService {
          rpc AddItem(AddItemRequest) returns (Empty) {}
          rpc GetCart(GetCartRequest) returns (Cart) {}
          rpc EmptyCart(EmptyCartRequest) returns (Empty) {}
      }

      message CartItem {
          string product_id = 1;
          int32  quantity = 2;
      }

      message AddItemRequest {
          string user_id = 1;
          CartItem item = 2;
      }

      message EmptyCartRequest {
          string user_id = 1;
      }

      message GetCartRequest {
          string user_id = 1;
      }

      message Cart {
          string user_id = 1;
          repeated CartItem items = 2;
      }

      message Empty {}

      // ---------------Recommendation service----------

      service RecommendationService {
        rpc ListRecommendations(ListRecommendationsRequest) returns (ListRecommendationsResponse){}
      }

      message ListRecommendationsRequest {
          string user_id = 1;
          repeated string product_ids = 2;
      }

      message ListRecommendationsResponse {
          repeated string product_ids = 1;
      }

      // ---------------Product Catalog----------------

      service ProductCatalogService {
          rpc ListProducts(Empty) returns (ListProductsResponse) {}
          rpc GetProduct(GetProductRequest) returns (Product) {}
          rpc SearchProducts(SearchProductsRequest) returns (SearchProductsResponse) {}
      }

      message Product {
          string id = 1;
          string name = 2;
          string description = 3;
          string picture = 4;
          Money price_usd = 5;

          // Categories such as "clothing" or "kitchen" that can be used to look up
          // other related products.
          repeated string categories = 6;
      }

      message ListProductsResponse {
          repeated Product products = 1;
      }

      message GetProductRequest {
          string id = 1;
      }

      message SearchProductsRequest {
          string query = 1;
      }

      message SearchProductsResponse {
          repeated Product results = 1;
      }

      // ---------------Shipping Service----------

      service ShippingService {
          rpc GetQuote(GetQuoteRequest) returns (GetQuoteResponse) {}
          rpc ShipOrder(ShipOrderRequest) returns (ShipOrderResponse) {}
      }

      message GetQuoteRequest {
          Address address = 1;
          repeated CartItem items = 2;
      }

      message GetQuoteResponse {
          Money cost_usd = 1;
      }

      message ShipOrderRequest {
          Address address = 1;
          repeated CartItem items = 2;
      }

      message ShipOrderResponse {
          string tracking_id = 1;
      }

      message Address {
          string street_address = 1;
          string city = 2;
          string state = 3;
          string country = 4;
          string zip_code = 5;
      }

      // -----------------Currency service-----------------

      service CurrencyService {
          rpc GetSupportedCurrencies(Empty) returns (GetSupportedCurrenciesResponse) {}
          rpc Convert(CurrencyConversionRequest) returns (Money) {}
      }

      // Represents an amount of money with its currency type.
      message Money {
          // The 3-letter currency code defined in ISO 4217.
          string currency_code = 1;

          // The whole units of the amount.
          // For example if `currencyCode` is `"USD"`, then 1 unit is one US dollar.
          int64 units = 2;

          // Number of nano (10^-9) units of the amount.
          // The value must be between -999,999,999 and +999,999,999 inclusive.
          // If `units` is positive, `nanos` must be positive or zero.
          // If `units` is zero, `nanos` can be positive, zero, or negative.
          // If `units` is negative, `nanos` must be negative or zero.
          // For example $-1.75 is represented as `units`=-1 and `nanos`=-750,000,000.
          int32 nanos = 3;
      }

      message GetSupportedCurrenciesResponse {
          // The 3-letter currency code defined in ISO 4217.
          repeated string currency_codes = 1;
      }

      message CurrencyConversionRequest {
          Money from = 1;

          // The 3-letter currency code defined in ISO 4217.
          string to_code = 2;
      }

      // -------------Payment service-----------------

      service PaymentService {
          rpc Charge(ChargeRequest) returns (ChargeResponse) {}
      }

      message CreditCardInfo {
          string credit_card_number = 1;
          int32 credit_card_cvv = 2;
          int32 credit_card_expiration_year = 3;
          int32 credit_card_expiration_month = 4;
      }

      message ChargeRequest {
          Money amount = 1;
          CreditCardInfo credit_card = 2;
      }

      message ChargeResponse {
          string transaction_id = 1;
      }

      // -------------Email service-----------------

      service EmailService {
          rpc SendOrderConfirmation(SendOrderConfirmationRequest) returns (Empty) {}
      }

      message OrderItem {
          CartItem item = 1;
          Money cost = 2;
      }

      message OrderResult {
          string   order_id = 1;
          string   shipping_tracking_id = 2;
          Money shipping_cost = 3;
          Address  shipping_address = 4;
          repeated OrderItem items = 5;
      }

      message SendOrderConfirmationRequest {
          string email = 1;
          OrderResult order = 2;
      }


      // -------------Checkout service-----------------

      service CheckoutService {
          rpc PlaceOrder(PlaceOrderRequest) returns (PlaceOrderResponse) {}
      }

      message PlaceOrderRequest {
          string user_id = 1;
          string user_currency = 2;

          Address address = 3;
          string email = 5;
          CreditCardInfo credit_card = 6;
      }

      message PlaceOrderResponse {
          OrderResult order = 1;
      }

      // ------------Ad service------------------

      service AdService {
          rpc GetAds(AdRequest) returns (AdResponse) {}
      }

      message AdRequest {
          // List of important key words from the current page describing the context.
          repeated string context_keys = 1;
      }

      message AdResponse {
          repeated Ad ads = 1;
      }

      message Ad {
          // url to redirect to when an ad is clicked.
          string redirect_url = 1;

          // short advertisement text to display.
          string text = 2;
      }

      // ------------Feature flag service------------------

      service FeatureFlagService {
        rpc GetFlag(GetFlagRequest) returns (GetFlagResponse) {}
        rpc CreateFlag(CreateFlagRequest) returns (CreateFlagResponse) {}
        rpc UpdateFlag(UpdateFlagRequest) returns (UpdateFlagResponse) {}
        rpc ListFlags(ListFlagsRequest) returns (ListFlagsResponse) {}
        rpc DeleteFlag(DeleteFlagRequest) returns (DeleteFlagResponse) {}
      }

      message Flag {
        string name = 1;
        string description = 2;
        bool enabled = 3;
        google.protobuf.Timestamp created_at = 4;
        google.protobuf.Timestamp updated_at = 5;
      }

      message GetFlagRequest {
        string name = 1;
      }

      message GetFlagResponse {
        Flag flag = 1;
      }

      message CreateFlagRequest {
        string name = 1;
        string description = 2;
        bool enabled = 3;
      }

      message CreateFlagResponse {
        Flag flag = 1;
      }

      message UpdateFlagRequest {
        string name = 1;
        bool enabled = 2;
      }

      message UpdateFlagResponse {}

      message ListFlagsRequest {}

      message ListFlagsResponse {
        repeated Flag flag = 1;
      }

      message DeleteFlagRequest {
        string name = 1;
      }

      message DeleteFlagResponse {}
    address: recommendationservice.service.consul:9001
    method: hipstershop.RecommendationService.ListRecommendations
specs:
 - selector: span[name="get_product_list"]
   assertions:
   - attr:app.filtered_products.count  =  9

Save this file as recsvc-test.yml in the tracetest/tests directory of nomad-conversions.

4- Run the Trace-based test

With our test YAML in hand, we can run our test! Make sure that you’re running this from tracetest directory of nomad-conversions:

tracetest test run --definition tests/recsvc-test.yml

Sample output:

✔ RecommendationService test (http://tracetest.localhost/test/avtest4567/run/1/test)

Now let’s check out our test in the Tracetest UI by going to http://tracetest.localhost:

Let’s click on the > next to the test definition to expand it:

Note that the UI shows that it ran v1 of the test. If we make any changes to the test definition, Tracetest bumps the test version; however, if we were to change only the test ID and re-run the file, it would show up as an entirely different test in Tracetest, and not a different version of the same test.

Let’s explore our test definition in the UI, by clicking in the area highlighted below:

Will take us to the test trigger definition screen:

This might seem familiar - it’s the information that we entered in our test YAML. Tracetest runs the API endpoint and then checks to see if there are any Traces associated with this with this endpoint, and lets you create tests against the traces. Test specs are defined in the Test tab, located at the top middle section of the screen:

Clicking on it brings us to the test spec definition screen:

Next, let’s head to the top left-hand side of the screen, and click on the purple circle with >>:

This reveals a side pane with all sorts of information about the selected Span. In our case, we’ve selected the Tracetest trigger Span (top box), so our screen looks something like this:

Notice how the Tracetest trigger box has a slight blue highlight. But we’re not interested in this Span. We’re actually interested in the get_product_list Span, so let’s click on that Span:

Notice how it has an attribute called app.filtered_products.count, and that its value is 9…which is the value that we’re checking for in our assertion! Ta-da! 🎉

Part 4: See it Lightstep!

Remember that our OTel Collector is configured to send Traces to both Tracetest and Lightstep, so we should see the same Trace data in both.

I went into my Lightstep account and created a Notebook that looked like this:

And clicked on one of the matching traces returned, which led me to the Trace diagram below:

As you can see, we have our get_product_list trace, which has an attribute called app.filtered_products.count with a value of 9. Yay!

Final Thoughts

Tracetest is a great tool for unlocking the potential of Trace-based testing. Although the example in this tutorial was fairly simple, I hope that it gives you an appreciation for some of the cool stuff that Tracetest can do. I definitely plan to keep my eye out as this tool improves, and I am excited to see where Trace-based testing takes us in 2023!

For more on the ins and outs of Tracetest, be sure to check out the official docs here. For you visual learners, I’ve put together a quick little four-minute video highlighting the main things covered in this post. Be sure to check it out here.

That was definitely a lot to take in, so give yourself a pat on the back for getting through this! Now please enjoy this lovely photo of Mookie the rat being cradled in my husband’s arms.

Peace, love, and code!

This post was originally published on the Tracetest blog.

Running Tracetest and the OpenTelemetry Demo App on HashiCorp Nomad

Adriana Villela — Wed, 18 Jan 2023 23:17:51 +0000

This quick little video gives a high-level overview of
running Tracetest and the OpenTelemetry (OTel) Demo App on HashiCorp Nomad. In this demo, I run tests against traces generated by a couple of gRPC services in the OTel Demo App.

I run Nomad locally using HashiQube.

How to Convert Kubernetes Manifests into Nomad Jobspecs

Adriana Villela — Mon, 19 Dec 2022 18:37:16 +0000

Ever since I started exploring Nomad, one of the things that I’ve enjoyed doing is taking Docker Compose files and Kubernetes manifests, and translating them into HashiCorp Nomad jobspec. I did it for Temporal back in March 2022, and also for an early version of Tracetest, back in the summer of 2022.

In my latest Nomadification Project (TM), I got the OpenTelemetry Demo App to run on Nomad (with HashiQube, of course). To do this, I used the OpenTelemetry Demo App Helm Chart as my guide. In doing this, and other Nomadifications, I realized that I’ve never gone through the process of explaining the conversion process from Kubernetes manifests to Nomad jobspecs.

So, as you may have guessed, today, I will go through the process of converting Kubernetes manifests to Nomad jobspecs, so that if you ever find yourself in a situation whereby you’re thinking, “Gee, it would be nice to see this Kubernetes stuff running on Nomad,” you now have a process!

I’ll use examples from the work I did recently in converting the OpenTelemetry Demo App Helm Charts into Nomad jobspecs to illustrate the process.

Are you ready? Let’s do this!

Manifests and Helm Charts and Jobspecs…oh my!

While I like working with both Kubernetes and Nomad alike, there is one thing that I find exceedingly irritating in Kubernetes Land: the fact that a Kubernetes manifest for an app deployment is made up of a scavenger hunt of YAML definitions of various Kubernetes objects. Nomad, however, takes a different approach, using a single HashiCorp Configuration Language (HCL) jobspec file as a one-stop shop for defining your app. I personally find Nomad HCL a lot easier to manage, since there are fewer moving parts, and when it comes to converting Kubernetes manifests to Nomad jobspecs, I find that having a single file to work with makes things a lot simpler.

In order to convert a Kubernetes manifest into a Nomad jobspec, we first need to start with a basic Nomad jobspec. This will serve as a template for deploying our application in Nomad.

Let’s start with our template jobspec below. Please bear in mind that this is a starting point for our conversion. After all, some services are more complex than others, so while for some services, we need all of the components below to be included in our jobspec, for other services, we may end up with a more pared down version of the jobspec.

job "<service_name>" {
  type        = "service"
  datacenters = ["dc1"]

  group "<service_name>" {
    count = 1

    network {
      mode = "host"

      port "<port_name>" {
        to = <port_number>
      }
    }

    service {
      name = "<service_name>"
      port = "<port_name>"
      tags = [<tags_here>]

      check {
        <service_check_here>
      }
    }


    task "<service_name>" {
      driver = "docker"

      config {
        image = "<image_name>"
        image_pull_timeout = "25m"
        args = [<args_go_here>]
        ports = ["<port_name>"]
      }

      restart {
        attempts = 10
        delay    = "15s"
        interval = "2m"
        mode     = "delay"
      }

      env {
          <env_vars_here>
      }      

      template {
        data = <<EOF
<env_vars_derived_from_consul>
EOF
        destination = "local/env"
        env         = true
      }

      resources {
        cpu    = 60
        memory = 650
      }

    }
  }
}

Great…so now we’ve got our jobspec template. Yay! But we need to fill in the blanks, don’t we? So...where do we start?

Since we’re going from Kubernetes to Nomad, we need to look at the application’s Kubernetes manifest. Fortunately, we can grab this info easily from the OTel Helm Charts Repo, which, as you may have guessed, has a Helm Chart for the OTel Demo App. It also contains the rendered YAML manifests available to us here.

The OpenTelemetry Demo App is made up of a number of services. The process of converting the Kubernetes manifest of each service to its corresponding Nomad jobspec is very similar, so in the interest of not boring you to death, I’ll be choosing one service to illustrate the conversion: the featureflagservice.

Conversion Process

With the Nomad jobspec template and Kubernetes manifest in hand, we are ready to begin the conversion!

NOTE: You can find the repo with all of the OpenTelemetry Demo App jobspec files here.

1- Grab the Kubernetes manifests

As I mentioned earlier, the rendered YAML manifests for the OpenTelemetry Demo App are available to us here. Since, for the purposes of this tutorial, we only care about the featureflagservice’s Kubernetes manifest, I’ve gone ahead and grabbed the manifest pertaining to the featureflagservice, which is made up of a Deployment and a Service, as per below.

Here is the Deployment YAML:

---
# Source: opentelemetry-demo/templates/component.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-featureflagservice
  labels:
    helm.sh/chart: opentelemetry-demo-0.14.3
    app.kubernetes.io/name: example
    app.kubernetes.io/instance: example
    app.kubernetes.io/component: featureflagservice
    app.kubernetes.io/version: "1.2.1"
    app.kubernetes.io/part-of: opentelemetry-demo
    app.kubernetes.io/managed-by: Helm
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: example
      app.kubernetes.io/instance: example
      app.kubernetes.io/component: featureflagservice
  template:
    metadata:
      labels:
        app.kubernetes.io/name: example
        app.kubernetes.io/instance: example
        app.kubernetes.io/component: featureflagservice
    spec:
      containers:
        - name: featureflagservice
          image: 'ghcr.io/open-telemetry/demo:v1.2.1-featureflagservice'
          imagePullPolicy: IfNotPresent
          ports:

          - containerPort: 50053
            name: grpc
          - containerPort: 8081
            name: http
          env:
          - name: OTEL_SERVICE_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.labels['app.kubernetes.io/component']
          - name: OTEL_K8S_NAMESPACE
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
          - name: OTEL_K8S_NODE_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: spec.nodeName
          - name: OTEL_K8S_POD_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.name
          - name: FEATURE_FLAG_GRPC_SERVICE_PORT
            value: "50053"
          - name: FEATURE_FLAG_SERVICE_PORT
            value: "8081"
          - name: OTEL_EXPORTER_OTLP_TRACES_PROTOCOL
            value: grpc
          - name: DATABASE_URL
            value: ecto://ffs:ffs@example-ffspostgres:5432/ffs
          - name: OTEL_EXPORTER_OTLP_ENDPOINT
            value: http://example-otelcol:4317
          - name: OTEL_RESOURCE_ATTRIBUTES
            value: service.name=$(OTEL_SERVICE_NAME),k8s.namespace.name=$(OTEL_K8S_NAMESPACE),k8s.node.name=$(OTEL_K8S_NODE_NAME),k8s.pod.name=$(OTEL_K8S_POD_NAME)
          resources:
            limits:
              memory: 175Mi

Here is the Service YAML:

---
# Source: opentelemetry-demo/templates/component.yaml
apiVersion: v1
kind: Service
metadata:
  name: example-featureflagservice
  labels:
    helm.sh/chart: opentelemetry-demo-0.14.3
    app.kubernetes.io/name: example
    app.kubernetes.io/instance: example
    app.kubernetes.io/component: featureflagservice
    app.kubernetes.io/version: "1.2.1"
    app.kubernetes.io/part-of: opentelemetry-demo
    app.kubernetes.io/managed-by: Helm
spec:
  type: ClusterIP
  ports:
    - port: 50053
      name: grpc
      targetPort: 50053
    - port: 8081
      name: http
      targetPort: 8081
  selector:
    app.kubernetes.io/name: example
    app.kubernetes.io/instance: example
    app.kubernetes.io/component: featureflagservice

Yikes! This all looks pretty overwhelming, doesn’t it? Fortunately, it’s not as scary as it looks. Don’t worry…I’ll guide you along. Let’s keep going.

2- Prepare the jobspec

With our Kubernetes YAMLs in hand, let’s go back to our jobspec template from earlier, and fill in some blanks. Since we know that we’re working with the featureflagservice, I’ve gone ahead and replaced <service_name> with featureflagservice, which means that now our template looks like this:

job "featureflagservice" {
  type        = "service"
  datacenters = ["dc1"]

  group "featureflagservice" {
    count = 1

    network {
      mode = "host"

      port "<port_name>" {
        to = <port_number>
      }
    }

    service {
      name = "<service_name>"
      port = "<port_name>"
      tags = [<tags_here>]

      check {
        <service_check_here>
      }
    }


    task "featureflagservice" {
      driver = "docker"

      config {
        image = "<image_name>"
        image_pull_timeout = "25m"
        args = [<args_go_here>]
        entrypoint = [<entrypoints_go_here>]
        ports = ["<port_name>"]
      }

      restart {
        attempts = 10
        delay    = "15s"
        interval = "2m"
        mode     = "delay"
      }

      env {
          <env_vars_here>
      }      

      template {
        data = <<EOF
<env_vars_derived_from_consul>
EOF
        destination = "local/env"
        env         = true
      }

      resources {
        cpu    = 60
        memory = 650
      }

    }
  }
}

NOTE: You could technically give different names to your job, task and group, such as featureflagservice-job, featureflagservice-task and featureflagservice-group (or really anything you want), but for the sake of simplicity (with a sprinkling of lack of originality), I decided to give them all the same name: featureflagservice.

Some useful terminology:

job is the unit of control. The job is the thing that you start, stop, and update.
group is the unit of scale. The group defines how many instances you are running.
task is the unit of work. The task is what you actually want to run.

3- Port definitions

The next set of blanks that we need to fill in are in the network stanza. More specifically, the <port_name> and <port_number> values in the port stanza.

If we look at the featureflagservice’s Service YAML above, you’ll notice that it exposes two ports: 50053 (gRPC) and 8081 (HTTP), per spec -> ports -> targetPort. Let’s plug these into our jobspec:

network {
  mode = "host"

  port "http" {
    to = 8081
  }
  port "grpc" {
    to = 50053
  }
}

As you can see per the snippet above, we labeled (named) our ports http and grpc. These labels will allow us to refer to those ports by a human-friendly label, rather than by number. Which means that if one or both of the port numbers change, we only need to make the change in one place. And spoiler alert: we will be referring to those ports elsewhere in the jobspec.

NOTE: Feel free to label your ports anything you want–just make sure that it’s reasonably descriptive.

4- Service Definition

Now that we’ve defined our ports, we need to register our services, which is done by way of the service stanza. Since we have two ports in the network stanza above, we need to define two services: one per port.

The service definition for the http port looks like this:

service {
  name = "featureflagservice-http"
  port = "http"
  tags = [
    "traefik.http.routers.otel-collector-http.rule=Host(`featureflag.localhost`)",
    "traefik.http.routers.otel-collector-http.entrypoints=web",
    "traefik.http.routers.otel-collector-http.tls=false",
    "traefik.enable=true",
  ]

  check {
    type     = "tcp"
    interval = "10s"
    timeout  = "5s"
  }
}

Noteworthy items:

By default, the service is registered to Consul. Although we don’t explicitly say so, it’s the equivalent of adding a provider = "consul" attribute to the service stanza. You can register your services to either Nomad or Consul.
The port attribute is the network port label to which the service applies.
The service is called featureflagservice-http. Again, you can call it whatever you want, though a descriptive name is always helpful.
We’re exposing this service to the outside world via Traefik, and the service is accessible via the URL: http://featureflag.localhost (since I’m running this locally, using HashiQube). Keep in mind that you also need to deploy the Traefik jobspec alongside the featureflagservice jobspec in order to expose this service to the outside world. To learn more about running Traefik on Nomad, check out this post and this post.
The check stanza runs a health check on the service Since the service is registered to Consul, the health check runs on Consul. The above health check is configured to run every 10 seconds, and is given 5 seconds for the health check query to succeed. Health checks in Nomad are similar to Kubernetes liveness probes. Setting the on_update attribute creates something closer to a Kubernetes readiness probe.

The service for the grpc port looks like this:

service {
  name = "featureflagservice-grpc"
  port = "grpc"

  check {
    type     = "tcp"
    interval = "10s"
    timeout  = "5s"
  }
}

Noteworthy items:

Since we’re not exposing any outside services, we don’t need the tags attribute with the Traefik configurations.
The port attribute refers to the grpc port that we defined in the network stanza earlier.
We’re doing the same health check that we did for the http port.

For additional examples health checks, check out:

A gRPC health check in the recommendationservice jobspec
A command-based health check to check database connectivity in the ffspostgres jobspec

5- Task Definition

Okay…now we’re ready to define our task. Since we’re running a containerized workload, our task uses the Docker driver.

Config Stanza

Since we’re using the Docker driver, we need to provide the following information to Nomad via the config stanza:

Name of the Docker image. We get this information from spec -> template -> spec -> containers -> image in the Deployment YAML. In this case, the image name is ghcr.io/open-telemetry/demo:v1.2.1-featureflagservice.
Ports being used by the Docker image. We get this information from spec -> template -> spec -> containers -> ports in the Deployment YAML. In this case, the image requires ports 50053 and 8081, which we named http and grpc, respectively, in our jobspec’s network stanza

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-featureflagservice
...
spec:
...
    spec:
      containers:
        - name: featureflagservice
          image: 'ghcr.io/open-telemetry/demo:v1.2.1-featureflagservice'
...

This translates to the config stanza of the featureflagservice task looking like this:

config {
  image = "ghcr.io/open-telemetry/demo:v1.2.1-featureflagservice"
  image_pull_timeout = "25m"
  ports = ["http", "grpc"]
}

A few noteworthy items:

Since there are no args present in the Deployment YAML, we’re omitting args from this jobspec. If you’d like to see an example of a jobspec that uses args, check out the Prometheus jobspec, and its corresponding Deployment YAML.
Since there is no command present in the Deployment YAML, we’re omitting the entrypoint from this jobspec. If you’d like to see an example of a jobspec that uses an entrypoint, check out the OTel Collector jobspec, and its corresponding Deployment YAML.
image_pull_timeout is set to 25 minutes. This is an optional value, and if you leave it out, it defaults to 5 minutes. I set it to a high value because sometimes you just never know if your network decides to give you the finger, and I don’t want the job to fail because it wasn’t able to pull the image within the allotted time.

Env Stanza

We’re not quite done with configuring our featureflagservice task. If you look at the Deployment YAML, you’ll notice that there are a number of environment variables under the env tag:

env:
  - name: OTEL_SERVICE_NAME
    valueFrom:
      fieldRef:
        apiVersion: v1
        fieldPath: metadata.labels['app.kubernetes.io/component']
  - name: OTEL_K8S_NAMESPACE
    valueFrom:
      fieldRef:
        apiVersion: v1
        fieldPath: metadata.namespace
  - name: OTEL_K8S_NODE_NAME
    valueFrom:
      fieldRef:
        apiVersion: v1
        fieldPath: spec.nodeName
  - name: OTEL_K8S_POD_NAME
    valueFrom:
      fieldRef:
        apiVersion: v1
        fieldPath: metadata.name
  - name: FEATURE_FLAG_GRPC_SERVICE_PORT
    value: "50053"
  - name: FEATURE_FLAG_SERVICE_PORT
    value: "8081"
  - name: OTEL_EXPORTER_OTLP_TRACES_PROTOCOL
    value: grpc
  - name: DATABASE_URL
    value: ecto://ffs:ffs@example-ffspostgres:5432/ffs
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: http://example-otelcol:4317
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: service.name=$(OTEL_SERVICE_NAME),k8s.namespace.name=$(OTEL_K8S_NAMESPACE),k8s.node.name=$(OTEL_K8S_NODE_NAME),k8s.pod.name=$(OTEL_K8S_POD_NAME)

You can ignore the ones that start with OTEL_K8S_, as they are Kubernetes-specific; however we do care about these:

OTEL_SERVICE_NAME
FEATURE_FLAG_GRPC_SERVICE_PORT
FEATURE_FLAG_SERVICE_PORT
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL
DATABASE_URL
OTEL_EXPORTER_OTLP_ENDPOINT
OTEL_RESOURCE_ATTRIBUTES

So how do we configure these in Nomad? Through the task’s env stanza. Which means that our environment variables look like this:

env {
  FEATURE_FLAG_GRPC_SERVICE_PORT = "${NOMAD_PORT_grpc}"
  FEATURE_FLAG_SERVICE_PATH_ROOT = "\"/feature\""
  FEATURE_FLAG_SERVICE_PORT = "${NOMAD_PORT_http}"
  OTEL_EXPORTER_OTLP_TRACES_PROTOCOL = "grpc"
  OTEL_RESOURCE_ATTRIBUTES = "service.name=featureflagservice"
}

A few noteworthy items:

Rather than hard-coding the value of FEATURE_FLAG_GRPC_SERVICE_PORT to 50053 and 8081, we’re using NOMAD_PORT_grpc and NOMAD_PORT_http. These are actually runtime environment variable, which follow the NOMAD_PORT_<label> naming convention. This prevents you from hard-coding the port number, which comes in handy if the port number changes in the network stanza for whatever reason, as you only need to change the number in one spot.
If you look at the Deployment YAML, you’ll notice that OTEL_RESOURCE_ATTRIBUTES is set to service.name=$(OTEL_SERVICE_NAME),k8s.namespace.name=$(OTEL_K8S_NAMESPACE),k8s.node.name=$(OTEL_K8S_NODE_NAME),k8s.pod.name=$(OTEL_K8S_POD_NAME). But I only set OTEL_RESOURCE_ATTRIBUTES to service.name=featureflagservice. Why? Well, because the other attributes in the Deployment YAML were Kubernetes-related, so I left them out.

Template Stanza

Wait…but why are DATABASE_URL and OTEL_EXPORTER_OTLP_ENDPOINT missing?? Well, if you look at the Deployment YAML, you’ll notice that the values for the above two environment variables are ecto://ffs:ffs@example-ffspostgres:5432/ffs and http://example-otelcol:4317, respectively.

Which begs the question...how does this translate to Nomad-speak? example-ffspostgres and example-otelcol, are the service names in Kubernetes for PostgreSQL and the OpenTelemetry Collector, respectively, so if we tried to use those same names in our jobspec definition, you’d get a big ‘ole nasty error from Nomad.

We could use the IP addresses of those services, but that’s not such a great idea, because IP addresses for services are bound to change, so if and when that address changes, your jobspec will fail to deploy.

What we need is a way to dynamically get a service’s IP address, given the service’s name. This is where Consul comes in. Among other things, Consul offers service discovery, which does exactly what we need.

To use Consul service-discovery, we need the following:

The name of the service that we’re referencing
The Nomad template stanza

The Nomad template stanza is very reminiscent of a Kubernetes ConfigMap. Per the Nomad docs, templates let you “ship configuration files that are populated from environment variables, Consul data, Vault secrets, or just general configurations within a Nomad task.” In our case, we’re using a template to query Consul services, so that we can find the IP address and port number of these services, so that we can populate our to populate our two remaining environment variables, DATABASE_URL and OTEL_EXPORTER_OTLP_ENDPOINT. The code for that looks like this:

template {
  data = <<EOF
{{ range service "ffspostgres-service" }}
DATABASE_URL = "ecto://ffs:ffs@{{ .Address }}:{{ .Port }}/ffs"
{{ end }}

{{ range service "otelcol-grpc" }}
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT = "http://{{ .Address }}:{{ .Port }}"
{{ end }}
EOF
  destination = "local/env"
  env         = true
}

Noteworthy items:

The template stanza is defined inside the task stanza.
The lines destination = "local/env" and env = true tell Nomad that these are environment variables
The line {{ range service "ffspostgres-service" }} tells Nomad to look for a service in Consul called ffspostgres-service. Once it finds the service name, we can pull the service’s IP address and port number using {{ .Address }} and {{ .Port }}, respectively.
Similarly, the line {{ range service "otelcol-grpc" }} tells Nomad to look for a service called otelcol-grpc. Once it finds the service name, we can pull the service’s IP address and port number using {{ .Address }} and {{ .Port }}, respectively.

But wait...where the heck do these service names come from?? Well, remember when we defined services in step 4 above, we gave each of our services a name?

ffspostgres-service is the name of the PostgreSQL service. You can check out the Nomad service definition here. (Aside: Take note of the service’s command-based health check to check database connectivity.)

Similarly, otelcol-grpc is the name of the gRPC service of the OpenTelemetry Collector. You can check out the service definition here.

For more info on Consul service discovery, check out this HashiCorp discussion forum. In addition, Nomad now has native service discovery sans Consul. For more info, check out docs here.

For an example of using the template stanza for configuration files, check out the OpenTelemetry Collector’s jobspec here.

Restart Rules

Unlike Docker Compose, you can’t specify dependency between services in Nomad. So, in order to ensure that Service X doesn’t die on you because it’s dependent on Service Y, which hasn’t started yet, you can put a restart policy into place. Below is the restart policy that I configured for featureflagservice:

restart {
  attempts = 10
  delay    = "15s"
  interval = "2m"
  mode     = "delay"
}

The above restart policy states that Nomad will try to restart the job 10 times in the span of 2 minutes. It will wait 15 seconds between restarts. If, after 10 attempts at a restart. By default, if the job still hasn’t started successfully, Nomad will fail the deployment and the job will be dead. This is dictated by the mode attribute, which defaults to fail. That’s not what we want, so instead we must set our mode to delay. This tells Nomad to restart the job another 10 times. This cycle continues on until the job finally starts up successfully.

Resource Allocations

If you follow my writings on Nomad, you’ll know that I am a HUGE fan of using HashiQube for running a Hashi environment on my local machine. This, of course, means that I have way less computing power than if I was, say, running this in Nomad in a datacenter. Which means that I have to be very mindful of the resources that I use, both for CPU and memory.

To get the correct values for CPU and memory usage, I had to play around a little. First, I started by deploying the jobspecs without any resource allocations, and checked out the jobs in Nomad to see if I either over-allocated or under-allocated resources.

For memory utilization, I looked at the resources consumed under the service’s allocation dashboard:

If you look at the above screen capture for the featureflagservice, you can see that I’m using about 60% of the memory that I allocated to this jobspec, which is pretty decent. If I deploy a service and see that it’s getting close to 100% memory usage (anything at 80% or above), I bump up the amount of memory used.

If you prefer the command line, you can run:

export ALLOCATION_ID=$(nomad job allocs -json featureflagservice | jq -r '.[0].ID')
nomad alloc status -stats $ALLOCATION_ID

Sample output:

...
Task "featureflagservice" is "running"
Task Resources:
CPU       Memory           Disk     Addresses
0/55 MHz  151 MiB/250 MiB  300 MiB  

Memory Stats
Cache  Swap  Usage
0 B    0 B   151 MiB

CPU Stats
Percent  Throttled Periods  Throttled Time
2.89%    0                  0
...

As you can see from the printout above CPU utilization is at 0 out of 55 MHz, and memory utilization is at 151 MiB out of 250 MiB.

For CPU utilization, I look at Nomad’s Topology dashboard.

I can see that for all of my services, I am using a grand total of 1.21 GHz of CPU for all of my jobspecs (all OTel Demo App jobspecs), out of my allotted 2 GHz (if you’re curious, I configured this setting here in HashiQube). By looking at my service’s CPU utilization from the allocation’s Resource Utilization dashboard, and by looking at how much compute power I have from the Topology dashboard, I can play around with the CPU utilization to reach a value that won’t exhaust my allocated resources. As a general rule of thumb, I like to make sure that all of my services are using 60-75% of the allotted resources.

So, with all that in mind, below are my resources settings for the featureflagservice, where CPU is measured in GHz, and memory is measured in MiB (mebibytes).

resources {
  cpu    = 55
  memory = 250
}

6- The Final Product!

Now that we’ve got all of our pieces in place, our final jobspec looks like this:

job "featureflagservice" {
  type        = "service"
  datacenters = ["dc1"]

  group "featureflagservice" {
    count = 1

    network {
      mode = "host"

      port "http" {
        to = 8081
      }
      port "grpc" {
        to = 50053
      }
    }

    service {
      name = "featureflagservice-http"
      port = "http"
      tags = [
        "traefik.http.routers.featureflagservice.rule=Host(`feature.localhost`)",
        "traefik.http.routers.featureflagservice.entrypoints=web",
        "traefik.http.routers.featureflagservice.tls=false",
        "traefik.enable=true",
      ]

      check {
        type     = "tcp"
        interval = "10s"
        timeout  = "5s"
      }
    }

    service {
      name = "featureflagservice-grpc"
      port = "grpc"

      check {
        type     = "tcp"
        interval = "10s"
        timeout  = "5s"
      }
    }

    task "featureflagservice" {
      driver = "docker"

      config {
        image = "otel/demo:v1.1.0-featureflagservice"
        image_pull_timeout = "10m"
        ports = ["http", "grpc"]
      }

      restart {
        attempts = 10
        delay    = "15s"
        interval = "2m"
        mode     = "delay"
      }

      env {
        FEATURE_FLAG_GRPC_SERVICE_PORT = "${NOMAD_PORT_grpc}"
        FEATURE_FLAG_SERVICE_PATH_ROOT = "\"/feature\""
        FEATURE_FLAG_SERVICE_PORT = "${NOMAD_PORT_http}"
        OTEL_EXPORTER_OTLP_TRACES_PROTOCOL = "grpc"
        OTEL_SERVICE_NAME = "featureflagservice"
      }

      template {
        data = <<EOF
{{ range service "ffspostgres-service" }}
DATABASE_URL = "ecto://ffs:ffs@{{ .Address }}:{{ .Port }}/ffs"
{{ end }}

{{ range service "otelcol-grpc" }}
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT = "http://{{ .Address }}:{{ .Port }}"
{{ end }}
EOF
        destination = "local/env"
        env         = true
      }

      resources {
        cpu    = 55
        memory = 250
      }

    }
  }
}

Ta-da!! 🎉

Final Thoughts

Whew! We covered a lot today! At the end of the day, I hope that this shows you that converting a Kubernetes manifest to a Nomad jobspec is not rocket science! It just takes a little bit of knowledge and patience.

Although this was by no means an exhaustive conversion, I hope that this little tutorial has given you the confidence to go from, “I wish that there was an example of how to run this on Nomad,” to, “I can get this to run in Nomad myself!”

I shall now reward you with a picture of Phoebe and our dearly departed Bunny, peering out of their cage.

Peace, love, and code. 🦄 🌈 💫

Got questions about Observability and/or OpenTelemetry? Want to collaborate on the OTel Demo App for Nomad? Talk to me! Feel free to connect through e-mail, or hit me up on Mastodon or LinkedIn. Hope to hear from y’all!

Running the OpenTelemetry Demo App on HashiCorp Nomad

Adriana Villela — Thu, 15 Dec 2022 23:59:12 +0000

Y’all...I’m so excited, because I finally got to work on an item on my tech bucket list. Last week, I began the process of translating OpenTelemetry (OTel) Demo App’s Helm Charts to HashiCorp Nomad job specs.

Today I’ll be talking about how to run the OpenTelemetry Demo App on Nomad, using my favorite Hashi-in-a-box tool, HashiQube.

Let’s do this!

Deployment

Assumptions

Before we move on, I am assuming that you have a basic understanding of:

Nomad. If not, head on over to my Nomad intro post. This blog post by Daniela Baron is also great.
Observability (o11y) and OpenTelemetry (OTel). If not, check out this Observability primer

Pre-Requisites

In order to run the example in this tutorial, you’ll need the following:

Docker (version 20.10.21 at the time of this writing)
Vagrant (version 2.3.1 at the time of this writing)

Tutorial Repos

Below are the repos that we’ll be using for today’s tutorial:

My modified HashiQube Repo (fork of servian/hashiqube). If you’re curious, you can see what modifications I’ve made here.
My Nomad Conversions repo

HashiQube Setup

Before you start, just a friendly reminder that HashiQube by default runs Nomad, Vault, and Consul on Docker. In addition, we’ll be deploying 21 job specs to Nomad. This means that we’ll need a decent amount of CPU and RAM, so
please make sure that you have enough resources allocated in your Docker desktop. For reference, I’m running an M1 Macbook Pro with 8 cores and 32 GB RAM. My Docker Desktop Resource settings are as follows:

CPUs: 3
Memory: 9.5GB
Swap: 3GB

Here’s a screenshot of my Docker Preferences Resources settings, if you need a visual:

For more, check out the Docker docs on how to change your resources settings for Mac, Windows, and Linux.

1- Update /etc/hosts

We use the Traefik load-balancer to expose our services, which we access as subdomains of localhost. In order ensure that we can access our Traefik-exposed services (and also the Traefik dashboard itself, you’ll need to add the following entries to /etc/hosts on your host machine:

127.0.0.1   traefik.localhost
127.0.0.1   otel-demo.localhost

2- Provision a Local Hashi Environment with HashiQube

Start HashiQube by following the detailed instructions
here.

NOTE: Be sure to check out the Gotchas section, if you get stuck.

Once everything is up and running (this will take several minutes, by the way), you’ll see this in the tail-end of the startup sequence, to indicate that you are good to go:

You can now access the apps using the URLs below:

Vault: http://localhost:8200

Nomad: http://localhost:4646

Consul: http://localhost:8500

Traefik: http://traefik.localhost

Don’t forget to download and install the
Nomad CLI
and the Vault CLI.

If you need to SSH into HashiQube, open up a new terminal window on your host machine and run the following command:

vagrant ssh

3- Deploy the OTel Demo App

We’re finally ready to deploy the OTel Demo App!

First, let’s clone the repo, and go to our working directory:

git clone https://github.com/avillela/nomad-conversions.git
cd nomad-conversions

Next, let’s enable
Memory Oversubscription in Nomad. This is a one-time setting.

nomad operator scheduler set-config -memory-oversubscription true

Memory Oversubscription allows Nomad to use more memory than is allotted to the job. For example, consider this setting in the resources stanza:

resources {
   cpu    = 55
   memory = 1024
   memory_max = 2048
}

We’ve allocated 55Mz of processing power to our job (cpu setting), along with 1024MB RAM (memory setting). In this case, when Memory Oversubscription is enabled, and the job requires more memory than the allotted 1024MB, Nomad will allocate as much as 2048MB RAM to the job (memory_max setting). Note that if Memory Oversubscription is not enabled,Nomad will ignore the memory_max setting.

Next, let’s deploy the services:

nomad job run -detach otel-demo-app/jobspec/traefik.nomad
nomad job run -detach otel-demo-app/jobspec/redis.nomad
nomad job run -detach otel-demo-app/jobspec/ffspostgres.nomad
nomad job run -detach otel-demo-app/jobspec/otel-collector.nomad
nomad job run -detach otel-demo-app/jobspec/adservice.nomad
nomad job run -detach otel-demo-app/jobspec/cartservice.nomad
nomad job run -detach otel-demo-app/jobspec/currencyservice.nomad
nomad job run -detach otel-demo-app/jobspec/emailservice.nomad
nomad job run -detach otel-demo-app/jobspec/featureflagservice.nomad
nomad job run -detach otel-demo-app/jobspec/paymentservice.nomad
nomad job run -detach otel-demo-app/jobspec/productcatalogservice.nomad
nomad job run -detach otel-demo-app/jobspec/quoteservice.nomad
nomad job run -detach otel-demo-app/jobspec/shippingservice.nomad
nomad job run -detach otel-demo-app/jobspec/checkoutservice.nomad
nomad job run -detach otel-demo-app/jobspec/recommendationservice.nomad
nomad job run -detach otel-demo-app/jobspec/frontend.nomad
nomad job run -detach otel-demo-app/jobspec/loadgenerator.nomad
nomad job run -detach otel-demo-app/jobspec/frontendproxy.nomad
nomad job run -detach otel-demo-app/jobspec/grafana.nomad
nomad job run -detach otel-demo-app/jobspec/jaeger.nomad
nomad job run -detach otel-demo-app/jobspec/prometheus.nomad

Since we’re running the jobs in detached mode, Nomad won’t wait to start the next job until the current one has deployed successfully. This means that your output will look something like this:

Job registration successful
Evaluation ID: d3eaa396-954e-241f-148d-6720c35f34bf
Job registration successful
Evaluation ID: 6bba875d-f415-36b7-bfeb-2ca4b9982acb
Job registration successful
Evaluation ID: 16dc8ef8-5e26-68f4-89b6-3d96b348775b
Job registration successful
Evaluation ID: 34de0532-a3b5-8691-bf18-51c0cc030573
Job registration successful
Evaluation ID: 7310e6a2-9945-710b-1505-c01bd58ccd35
...

A reminder that the Evaluation ID values will be different on your machine.

4- See it in Nomad!

As things are deploying, you can mozy on over to the Nomad UI at
http://localhost:4646 to see how things are coming along:

It will take some time for all of the services to come up (sometimes up to 10 minutes), especially since Nomad needs to download the images and initialize the services, so be patient! Since some services depend on other services in order to run, you may see services in limbo or some going up and down for a while, per the above screen capture. DON’T PANIC! IT WILL ALL BE OKAY!!

Once all of the jobs are up and running, you’ll see everything look green, like this:

You can also head on over to Consul at http://localhost:8500 to see the health of the services:

By default, unhealthy services show up at the top, with a red “x” next to them. Since we don’t see any nasty red “x”s in the above screen shot, we know that our services are lookin’ good!

5- Access the OTel Demo App

The OTel Demo App uses Envoy to expose a number of front-end services: the Webstore, Jaeger, Grafana, Load Generator, and Feature Flag. These are all managed by the frontendproxy service. Traefik makes the frontendproxy service available via the otel-demo.localhost address.

This is configured via the code snippet below, in the service stanza of frontendproxy.nomad:

tags = [        "traefik.http.routers.frontendproxy.rule=Host(`otel-demo.localhost`)",
    "traefik.http.routers.frontendproxy.entrypoints=web",
    "traefik.http.routers.frontendproxy.tls=false",
    "traefik.enable=true",
]

Note that the Host is set to otel-demo.localhost.

The services are accessed via the URLs below.

Webstore: http://otel-demo.localhost/

Go ahead and explore the amazing selection of telescopes and accessories, and buy a few. 😉🔭

Jaeger UI:
http://otel-demo.localhost/jaeger/ui/

In the screen capture above, we can see a sample Trace from the checkoutservice.

Grafana:
http://otel-demo.localhost/grafana/

The Demo App comes bundled with a two Grafana dashboards, which showcase Metrics emitted with OpenTelemetry.

Feature Flags UI:
http://otel-demo.localhost/feature/

Load Generator UI:
http://otel-demo.localhost/loadgen/

Gotchas

While I think I’ve managed to iron out a lot of the kinks as far as running the OTel Demo App in Nomad, I have run into a few hiccups when deploying the services.

Services sometimes can’t connect to the Collector

Although all of the services appear to start up properly, in some cases, some services appear to be unable to connect to the OTel Collector. I haven’t quite figured out why this is happening, so for now, I just restart otel-collector.nomad. If things are looking a little weird in the Webapp UI (like missing products or currency), I also restart frontend.nomad. Usually a good indicator that services aren’t sending telemetry to the Collector is to look at the number of services showing up in Jaeger. You should see 14 services, including the jaeger-query service.

Low memory on host machine

Yup…as beefy as my machine is, I do also sometimes run low on memory on my host machine. It probably doesn’t help that I have a zillion tabs open in Chrome and Safari. Plus, let’s face it: HashiQube + 21 jobs in Nomad can be a bit memory intensive. I’ve made a few tweaks to the memory settings in HashiQube and Docker to try to minimize memory issues, but in case the Memory Monster gets you, I suggest closing browsers and other apps, and re-opening them to free up some memory. And if this does happen to you, please let me know!

A Work in Progress

Please bear in mind that this project is a work in progress. If you have any suggestions for improvement, or would like to collaborate further on the Nomad jobspecs, please hit me up!

Final Thoughts

Well, there you have it, folks! You now have an example of how to deploy OpenTelemetry Demo App (a multi-micro-service app running OpenTelemetry) to HashiCorp Nomad. Main highlights:

We used HashiQube to stand up a local HashiCorp environment in Docker via Nomad so that we could run the OTel Demo App in Nomad using Traefik as our load balancer.
We saw the OTel Demo App in action, by accessing the following services exposed through the frontendproxy: Webstore, Grafana, Jaeger, Feature Flags UI,and the Load Generator UI.

Before I wrap this up, I do want to give a HUGE shoutout to Luiz Aoqui of HashiCorp, who helped me tweak my Nomad jobspecs, and to
Riaan Nolan, for his continued work on HashiQube.

Aside: Both Luiz and Riaan were my guests on the On-Call Me Maybe Podcast!

I will now leave you with a picture of Phoebe the rat, peering out of a pink basket. Doesn’t she look cute? 🥰

Peace, love, and code. 🦄 🌈 💫

Have questions about the OTel Demo App on Nomad? Feel free to connect through Mastodon or LinkedIn.

The OpenTelemetry community is always looking for contributions! Join us! If you're on Mastodon, be sure
to follow [OpenTelemetry on Mastodon (https://fosstodon.org/@opentelemetry)

Three Terraform Mistakes, and How to Avoid Them

Adriana Villela — Mon, 21 Nov 2022 15:56:17 +0000

In my last blog post, I talked about how Ana Margarita Medina and I used Terraform to show off Observability-Landscape-as-Code in practice, leveraging the OpenTelemetry Demo App to do so. The Demo App showcases instrumentation of Traces and Metrics of different services written in different languages using OpenTelemetry (OTel). Our Terraform code did the following:

Created a Kubernetes cluster
Deployed the Demo App to Kubernetes
Deployed OpenTelemetry Collector to Kubernetes, and configured it to send Traces and Metrics to Lightstep
Created dashboards in Lightstep

Now, I’m a fan of beautiful code, so we organized our code using Terraform Modules. We used a module for provisioning the Kubernetes cluster, one for deploying the OTel Demo App and the OTel Collector, and one for creating the Lightstep dashboards.

We also leveraged the following Terraform Providers:

Google Cloud Provider for spinning up a Kubernetes Cluster in Google Cloud Platform (GCP)
The Kubernetes Provider and the Helm Provider for deploying the app to Kubernetes
The Lightstep Provider for creating the dashboards in Lightstep

All good, right? Except for one teeeensy little problem...the last time I’d touched Terraform was in early 2021, and even then, I was just tweaking code. So I kinda had to teach myself Terraform all over again. And I hit up a few snags along the way. Cue. The. Panic.

Fortunately, Google came through, and we were able to resolve the issues. In today’s blog post, I will cover THREE Terraform gotchas that Ana and I hit, and how we solved them, so that you will hopefully be spared our utter despair and panic. 😅

Let’s do this!

NOTE: If you want to follow along to see the full Terraform source code, you can check it out here. Even though the source code is specific to the Observability-Landscape-as-Code use case, the main Terraform concepts in this blog post can be ported over to other scenarios.

Gotcha #1: The Chicken-and-Egg Scenario

After creating a Kubernetes cluster, we needed to create a Kubernetes resource before we could apply the Helm chart to install the OpenTelemetry demo app. The Demo App’s Helm Chart deploys an OpenTelemetry Collector. We wanted to configure the Collector to send OTel data to Lightstep. To do so, you need to add a Lightstep Access Token, which is stored as a Kubernetes secret.

You can learn more about the specifics of this setup here.

To create the secret in Kubernetes before running the Helm Chart, we used the Kubernetes Provider. In order to use this provider, Terraform needs to know information about your cluster, so that it knows what cluster to apply the manifest to. To do this, I needed to store the cluster information in the data stanza, like this:

data "google_client_config" "default" {}

data "google_container_cluster" "primary" {
 name     = var.cluster_name
 location = var.region
}

provider "kubernetes" {
 host  = "https://${data.google_container_cluster.primary.endpoint}"
 token = data.google_client_config.default.access_token
 cluster_ca_certificate = base64decode(
   data.google_container_cluster.primary.master_auth.0.cluster_ca_certificate
 )
}

Easy peasey, right? Unfortunately, when I ran terraform apply, I kept getting the following errors:

Error: Invalid template interpolation value

And

Error: Attempt to index null value

Basically, Terraform was trying to evaluate the contents of the data stanza (which were null) before it had any information about the Kubernetes cluster. Which of course it didn’t, because the cluster didn’t yet exist!! Hence the null contents.

I frantically Googled this one for a while, spinning my wheels. And then, the “aha” moment hit me, when I saw somewhere in one of my searches that I could use the depends_on attribute in the data stanza. So I added depends_on = [module.k8s_cluster_create] to both my data stanzas, which basically says, “Hey buddy, don’t try to evaluate this until AFTER the k8s_cluster_create module (i.e. the module in which the Kubernetes cluster is created) is run. So now, after adding depends_on, my providers.tf (lines 32-49) code looked like this:

data "google_client_config" "default" {
 depends_on = [module.k8s_cluster_create]
}

data "google_container_cluster" "primary" {
 depends_on = [module.k8s_cluster_create]
 name     = var.cluster_name
 location = var.region
}

provider "kubernetes" {
 host  = "https://${data.google_container_cluster.primary.endpoint}"
 token = data.google_client_config.default.access_token
 cluster_ca_certificate = base64decode(
   data.google_container_cluster.primary.master_auth.0.cluster_ca_certificate
 )
}

And after making that change, all was well with the world. Huzzah!

Gotcha #2: Using Modules with depends_on

While the above problem went away, I then found myself face-to-face with yet another conundrum. When I initially wrote my Terraform code, everything was in one big file, and it worked just fine. So OF COURSE I just assumed that when I prettified my code and moved things into modules, I could just get away defining my Providers in the Modules themselves. Well, you can. That is…if you don’t use the depends_on attribute in your Module call.

So basically, when I tried to say that the Module lightstep_dashboards depended on k8s_cluster_create like this:

module "k8s_cluster_create" {
   source = "./modules/k8s"

   cluster_name = var.cluster_name
   project_id = var.project_id
   region = var.region
   network = var.network
   subnet = var.subnet
}

module "deploy_otel_demo_app" {
   source = "./modules/otel_demo_app"

   otel_demo_namespace = var.otel_demo_namespace
   ls_access_token = var.ls_access_token
   cluster_name = var.cluster_name
   project_id = var.project_id
   region = var.region
   network = var.network
   subnet = var.subnet
}

module "lightstep_dashboards" {
   source = "./modules/lightstep"
   depends_on = [module.k8s_cluster_create]

   lightstep_project = var.ls_project
}

I kept getting this error when I ran terraform apply:

Error: Module is incompatible with count, for_each and depends_on

This error happens when the Child Module contains a provider block and the Module that you’re trying to call is using count, depends_on, and/or for_each. Why? Because provider blocks inside a Child Module are not allowed when your Module call is using count, depends_on, and/or for_each. You can read up more on this here.

Well, it turns out that correct practice is to define your provider block in the Root Module, as Providers are automagically passed down to the Child Modules. So to make the above error go away, I moved all of my Provider definitions to the Root Module, and was able to keep depends_on in my Module call. If I didn’t have any dependencies, I could’ve left out the depends_on block, but I wouldn’t really be following the recommended practice.

NOTE: You can learn more about Providers and Modules here.

Gotcha #3: Referencing a non-TF provider in a module

Two problems down. Awesome! Unfortunately, my problems were not over. I continued to anger the Module Gods, because I encountered yet another issue when I moved my non-modularized code into Modules. This time, it had to do with using the Lightstep Provider. You see, this Provider comes from a third-party (i.e. not HashiCorp), which in this case is Lightstep. Lightstep is what is known as a Partner Provider. This means that in the Provider Registry, the Provider is named lightstep/lightstep, where the first lightstep means that the Provider is created and maintained by Lightstep, and the second lightstep is the actual Provider name. For comparison, the hashicorp/google provider is an Official Provider, because it is created and maintained by HashiCorp.

Now here’s the odd part. When I tried to run terraform init, I was graced with this error:

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/lightstep

Um...what? This did not compute, because in my providers.tf file, I CLEARLY said that the Provider name was lightstep/lightstep, so where oh where was it getting this hashicorp/lightstep business from?? LOOK ⬇️⬇️⬇️

terraform {
 required_providers {
...
   lightstep = {
     source = "lightstep/lightstep"
     version = ">=1.70.0"
   }
...
}

O Google gods, help meeeeee!!

Well, it turns out that when using a Partner Provider in a Module, Terraform assumes the Provider is an Official Provider, and is therefore automagically given a hashicorp suffix when passing it down to the Module. So Terraform basically thought that the Provider was called hashicorp/lightstep, even though I clearly defined it correctly in the Providers section of the Root Module.

To fix this issue, I ended up having to define a required_providers stanza in the Root Module, as I had already done, AND I also had to add a required_providers stanza to my Child Module, as per the snippet below:

terraform {
 required_providers {
   lightstep = {
     source = "lightstep/lightstep"
     version = ">=1.70.0"
   }
 }
}

After that, my terraform init stopped screaming at me!

Final Thoughts

Today we learned that Terraform can be a wee finicky. We learned that:

Adding depends_on to the data stanza used to capture your Kubernetes cluster configuration data ensures that Terraform doesn’t try to evaluate the data stanza until AFTER the cluster is created, thereby avoiding some serious Terraform Anger™.
If you want to use depends_on in a Module call, the Provider configuration must be done in the Root Module. Also, it’s the recommended practice even if you don’t want to use depends_on.
If you have a Module that references a Partner Provider, you need to define a required_providers stanza in both the Root Module and the Child Module.

I hope that these tips prevent you from experiencing Terraform Anguish™ next time you find yourself Terraformin’. And now, I shall reward you with a picture of my rat Mookie, who is seen below peering out of an authentic Wisconsin Cheese Head hat.

Peace, love, and code. 🦄 🌈 💫

Got questions about Terraform or Observability-Landscape-as-Code? Talk to me! Feel free to connect through e-mail, or hit me up on Twitter, Mastodon, or LinkedIn. Hope to hear from y’all!

Observability-Landscape-as-Code in Practice

Adriana Villela — Tue, 15 Nov 2022 15:47:02 +0000

with Ana Margarita Medina

If you follow Adriana's writings on Observability, you may recall a post from back in June introducing the concept of Observability-Landscape-as-Code (OLaC).

An Observability Landscape is made up of the following pieces:

Application instrumentation
Collecting and storing application telemetry
An Observability back-end
A set of meaningful SLOs
Alerts for on-call Engineers

Keeping that in mind, OLaC is simply the codification of your Observability Landscape, thereby ensuring consistency, maintainability, and reproducibility.

That’s all well and good, but how about seeing this thing in action? Well, my friend, you’ve come to the right place, because today, you get to see a tutorial featuring a number of OLaC practices in action!

Today, Ana Margarita Medina (my fellow DevRel and partner in crime) and I will be highlighting the following aspects of OLaC:

Application instrumentation

How? À la OpenTelemetry, via the OpenTelemetry Demo App, which contains examples of Traces and Metrics instrumentation.
Collecting & storing application telemetry

How: OpenTelemetry Collector is deployed via code (Helm chart), alongside the various services that make up the OpenTelemetry Demo App.
Codifying your Observability back-end configuration

How: Using the Lightstep Terraform Provider to create dashboards in Lightstep.

We wanted to showcase OLaC principles with a real-life example using modern cloud-native tooling...Which means using Kubernetes for our cloud infrastructure with Google Cloud’s Kubernetes offering, GKE. Now, since we are good practitioners of OLaC and SRE, we won’t just be setting things up through the clickity click of a UI. No sirreee. Instead, we’ll be #automatingAllTheThings using HashiCorp Terraform. Terraform allows us to do infrastructure-as-code (IaC), and gives us tons of added benefits like better control over our resources and standardization. These are key principles in OLaC and IaC.

We will be deploying OpenTelemetry Demo App to our cluster. The Demo App has been instrumented using OpenTelemetry, and will send Traces and Metrics through the OpenTelemetry Collector to Lightstep.

Are you ready??? Let’s get started!

Tutorial

Pre-Requisites

Before you begin, you will need the following:

A Lightstep account so you can see application Traces, and Metrics dashboards
A Lightstep Access Token for the Lightstep project you would like to use
A Lightstep API key for creating dashboards in Lightstep.
Terraform CLI to run the Terraform scripts
A Google Cloud account, so you can create a Kubernetes cluster (GKE)
gcloud CLI to interact with Google Cloud
kubectl to interact with Kubernetes

Steps

In today’s tutorial, we’ll be running Terraform code to showcase OLaC in action. For convenience, the main components have been broken up into separate Terraform modules that will do the following:

Create a Kubernetes cluster (GKE) in Google Cloud using the Google Terraform Provider. This is defined in the k8s module in our repo.
Deploy the OpenTelemetry Demo App to the cluster using the Helm Terraform Provider. This is defined otel_demo_app module in our repo.
Create Metrics dashboards in Lightstep Terraform Provider. This is defined in the lightstep module in our repo.

The full code listing for this tutorial can be found here.

1- Clone the example repo

Let's start by cloning the example repo:

git clone https://github.com/lightstep/unified-observability-k8s-kubecon.git

2- Initialize Sub-Modules

This project makes use of a few Git submodules, so in order to ensure that things work nicely, you’ll need to pull them in:

cd unified-observability-k8s-kubecon
git submodule init && git submodule update

3- Google Cloud Login

Before we can create a GKE cluster you must authenticate your Google Cloud account:

gcloud auth application-default login --no-launch-browser

You will be presented with a link which you need to open up in a browser, to authenticate your Google ID. Once you are authenticated, the browser will display an authorization token for you to paste in the command line, as follows:

4- Create terraform.tfvars

Now that you’re authenticated, let’s get ready to Terraform! Before you can do that, we need to create a terraform.tfvars file.

Lucky for you, we have a handy-dandy template that you can use to get started:

cd k8s-cluster-with-otel-demo/terraform
cp terraform.tfvars.template terraform.tfvars

Next populate the following values in the file:

<your_gcp_project>: The name of your Google Cloud project. Don’t know your project name? No problem! Just run gcloud config get-value project to find out what it is!
<your_gke_cluster_name>: The name you wish to give your GKE cluster. Make sure it follows Kubernetes cluster naming conventions (i.e. no underscores _ or special characters).
<your_lightstep_access_token>: Your Lightstep Access Token. This is used to send Traces to your Lightstep Project.
<your_lightstep_api_key>: Your Lightstep API key. This is used to create our Metrics dashboards.
<your_lightstep_org_name>: Your Lightstep organization name. Not sure what your organization is called? No problem! Log into Lightstep,and click on the person icon on the bottom left of your screen. This will pop up a little menu. The organization name can be found under the “Account Management” heading, like this:

Notice that my organization is called “LightStep”. Yours will be different. Note also that Organization names are case-sensitive.

<your_lightsep_project_name>: The name of the Lightstep Project where the Metrics dashboards will be displayed.

Note: terraform.tfvars is in .gitignore and won't be put into version control.

5- Run Terraform

This step will initialize Terraform (install providers locally), and then will apply the Terraform plan.

It will:

Create a Kubernetes cluster
Deploy the OpenTelemetry Demo App using the OpenTelemetry Demo Helm Chart
Create dashboards in Lightstep

Before running the commands below, make sure that you’re already in the k8s-cluster-with-otel-demo/terraform folder.

terraform init
terraform apply -auto-approve

Please note that this step may take up to 30 minutes, depending on GKE’s disposition. Be patient. 😄

6- Update your kubeconfig

Now that the cluster is created, you can add it to your kubeconfig file! By default, the file is saved at $HOME/.kube/config.

Before you can update your kubeconfig, you first need to make sure that you have the gke-gcloud-auth-plugin installed:

gcloud components install gke-gcloud-auth-plugin
gke-gcloud-auth-plugin --version
echo "export USE_GKE_GCLOUD_AUTH_PLUGIN=True" >> ~/.bashrc

Now we can add the cluster to kubeconfig:

gcloud container clusters get-credentials $(terraform output -raw kubernetes_cluster_name) --region $(terraform output -raw region)

This gets the kubernetes_cluster_name and region output values from Terraform (that’s the terraform output -raw stuff), and plunks those into your gcloud container clusters get-credentials command.

Or, if you closed the terminal in which you were running Terraform and lost your output values, you can also do this:

gcloud container clusters get-credentials <cluster_name> --region <region>

Where <cluster_name> and <region> correspond to the values you entered in Step 3 in your terraform.tfvars file.

7- Check out the OTel Demo app

Now that you’ve got the OpenTelemetry Demo App deployed to your cluster, let’s take a look at it! First, let’s peek into Kubernetes to see what’s up.

If you run kubectl get ns, you’ll notice that there’s now a new namespace called otel-demo:

This is where we deployed the OTel Demo app. Let’s look into this namespace to see what we’ve created. First, let’s look at the pods with kubectl get pods -ns otel-demo:

Notice how we deployed a bunch of different services that make up the OTel Demo App, including adservice, cartservice, recommendationservice, etc.

We also deployed an OTel Collector. Its configuration YAML is stored in a configmap. We can take a peek by running kubectl describe configmap otel-demo-app-otelcol -n otel-demo:

You can see that we also reference a variable called ${LS_TOKEN} which represents your Lightstep Access Token, which you set in terraform.tfvars. But where is it? The secret is mounted to the OTel Collector container instance as a secret called otel-collector-secret. Let’s take a look at the secret by running kubectl describe secret otel-collector-secret -n otel-demo:

All this magic happens in otel-demo-app-values-ls.yaml. This is a version of values.yaml from the OTel Demo App Helm Chart with updates to the Collector configs so that we can configure the OTel Collector to send Traces to Lightstep.

8 - Run the OTel Demo App

Okay...enough Kubernetes talk. Let’s look at the OpenTelemetry Demo App! You can access the Demo App by Kubernetes port-forward:

kubectl port-forward -n otel-demo svc/otel-demo-app-frontend 8080:8080

To access the front-end, go to http://localhost:8080:

Go ahead and explore the amazing selection of telescopes and accessories, and buy a few. 😉🔭

9- See Traces in Lightstep

We can now pop over to Lightstep and check out some Traces. Let’s do this by creating a Notebook.

First, click on the little page icon on the left nav bar (highlighted in blue, below). That will bring up this page:

Next, we build our query for our Traces. Let’s look at the traces from the recommendationservice. We’ll do by entering recommendationservice in the field next to “All telemetry”. Because this is a service, select the second value from the drop-down, which says, “Use ‘recommendationservice’ as service value”, as per below:

After you select that value, you’ll see a chart like this:

The little green dots represent trace exemplars from that Service. Hover over one of them to see for yourself!

If you click on one of these dots, you’ll get taken to the Trace view. Before you click, be sure to save your Notebook first (don’t worry, you’ll get a reminder before you navigate away from the page)!

Here’s the Trace view we see when we click on the get_product_list dot (Operation) above::

Pretty cool, amirite?

10- See Kubernetes Metrics in Lightstep

But wait...there’s more! Not only can we see Traces in Lightstep, we can also see Metrics!

Remember when you ran terraform apply? Well, not only did it create a Kubernetes cluster, deploy the OTel Demo App (and OTel Collector), it also created some handy-dandy Metrics dashboards for us.

You can check out the newly-created Metrics dashboards by going to the Dashboards icon (the icon with 4 little squares) on the left navigation bar:

First, let’s check out the Kubernetes / Compute Resources / Cluster dashboard. This dashboard lets you see the state of your cluster.

We then have various other Metrics called Kubernetes Workload Metrics. These are the dashboards with names that start with “Kubernetes / Compute Resources / Workload”. These dashboards are specific to the services you are running. They take into account the Kubernetes Workloads in your various namespaces, using kube-state-metrics. For a closer look, check out otel_demo_app_k8s_dashboard.tf.

We used Lightstep’s Prometheus Kubernetes OpenTelemetry Collector to get these Metrics into Lightstep. This Helm chart is inspired by kube-prometheus-stack, but with one crucial difference -- no Prometheus! We’re able to use recent enhancements to the OpenTelemetry Operator for Kubernetes such as support for Service Monitors in order to scrape Prometheus metrics from pods, system components, and more.

The best part is that you don’t need to install and maintain a Prometheus instance to be able to run it! But wait…there’s more! You also don’t need to use Lightstep as your Observability back-end in order to take advantage of this special Collector! How cool is that??

Note: You can learn more about the Prometheus Kubernetes OpenTelemetry Collector by checking out the docs here.

For example, the Kubernetes / Compute Resources / Workload / otel-demo-app-cartservice dashboard displays metrics for the OTel Demo App’s cartservice. In it we can see how our containers and pods are doing based on Metrics such as those for CPU and Memory.

11- See Application Metrics in Lightstep

Ah...but we’re not done with Metrics just yet! If you go back to the dashboard view and scroll to the very end of the list, you’ll see the OTel Demo App - Application Metrics dashboard.

Let’s click on it to take a quick little peek!

The latest version of the OTel Demo App emits both auto-instrumented and manually-instrumented Metrics. In today’s demo, we wanted to highlight some of the Metrics from the recommendationservice.

First, we have the auto-instrumented Python Metrics, which are captured from the Python runtime:

runtime.cpython.cpu_time: Track the amount of time being spent in different states of the CPU. This includes user (time running application code) and system (time spent in the operating system). This metric is represented as total elapsed time in seconds.
runtime.cpython.memory: Memory utilization
runtime.cpython.gc_count: Number of times the garbage collector has been called.

We also have one manually-instrumented Metric:

app_recommendations_counter: Cumulative count of the number of recommended products per service call

For more on the recommendationservice Metrics, check out this doc. For more on Metrics captured by other services, check out the OTel Demo App service docs.

12- Teardown

If you’re no longer using this environment, don’t forget to tear down its resources, to avoid running up a huge cloud bill. You’re welcome. 😉

terraform destroy -auto-approve

This step can take up to 30 minutes, so please be patient! Also, you’ll probably notice that on first run, you’ll see the following error:

Don’t panic! If you run terraform destroy -auto-approve again, it will finish nukifying all the things.

Final Thoughts

Today we got to see some aspects of Observability-Landscape-as-Code (OLaC) in practice! Specifically, we looked at the following elements:

Application instrumentation with OpenTelemetry
Collecting and storing application telemetry via the OTel Collector
Configuring an Observability back-end (i.e. Lightstep) through code

We showcased this by using Terraform to:

Deploy the OpenTelemetry Demo App to Kubernetes. The Otel Demo App showcases the Traces and Metrics instrumentation of different services in different languages using OpenTelemetry.
Deploy an OpenTelemetry Collector to Kubernetes (part of the Demo App deployment). The Collector is used to send application Traces and Metrics to Lightstep.
Configure Lightstep dashboards. The Lightstep Terraform provider allowed us to codify this.

Codifying our Observability Landscape means that we can tear down and recreate our application, Collector, and dashboards as needed, knowing that we’ll have consistency across the board every single time. Plus, it means that we can version control it, so that it’s not lost in the ether somewhere, or sitting in a secret server under Bob’s desk. Bonus!

Hopefully this gives you a nice little flavour of the power of OLaC, and will inspire you to go out there and start OLaC-ing too! (I just made up a new verb. You’re welcome.)

Whew! That was a lot to think about and take in! Give yourself a pat on the back, because we’ve covered a LOT! Now, please enjoy this picture of Adriana’s rat, Bunny, enjoying an almond!

Peace, love, and code. 🦄 🌈 💫

The OpenTelemetry Demo App is always looking for feedback and contributors. Please consider joining the OTel Community to help make OpenTelemetry AWESOME!

Got questions about Observability-Landscape-as-Code? Talk to us! Feel free to connect with us through e-mail, or:

Connect with Adriana on Twitter or LinkedIn
Connect with Ana on Twitter or LinkedIn.

Hope to hear from y’all!

KubeCon North America 2022: A Retrospective

Adriana Villela — Wed, 09 Nov 2022 20:15:53 +0000

with Ana Margarita Medina

Hello fellow Kubernetes nerds! 🤓 KubeCon North America is now behind us, so it’s time for a conference retrospective! Now, y’all must be thinking, “Awww, not another one! Noooo...” Ah, but ‘tis not just any KubeCon wrap-up, because both Ana and I will be sharing our perspectives, and we’re awesome and y’all love us. Amirite? Course I am. Without further ado...

Adriana’s Point-of-View

If there is any conference that is on My Conference Bucket List™, it is most definitely KubeCon. I have a love-hate relationship with Kubernetes (don’t we all, though?), and have spent hours trying to understand its wiley ways, and cursing at my terminal at yet another CrashLoopBackOff. So to go to KubeCon and nerd out on Kubernetes just sounded awesome to me.

I finally got my chance by attending KubeCon North America 2022, which took place in Detroit, MI.

I was super excited to attend the conference for a few reasons.

It would be the first time I’d be getting to meet my Lightstep DevRel team, and find out if I was the shortest of the bunch. Spoiler alert: Ana and I are the shortest at around 5’3 (161cm for my Metric peeps). Latina genes. I roll with it.
It was my conference début as a DevRel. Y’all, this is my first DevRel role, if you can believe it - 6 months in and lovin’ it!
Last time I attended a conference was DevOps Days Toronto, in 2018. The highlight of that was getting Dr. Nicole Forsgren and Jez Humble (both of DORA fame) to sign a hard copy of their book, Accelerate, so not too shabby. PS: Get the book. Bonus: if you get the audiobook, it’s read by Nicole.

So, enough rambling. Here’s my KubeCon recap, newbie edition!

As a COVID-cautious gal, I was pleased to see that the CNCF had various COVID precautions in place including vaccine requirement or a negative COVID test to pick your badge and masking was required and enforced. various COVID precautions in place. For starters, you needed proof of vaccination or proof of a negative COVID test in order to pick your badge. Inside the venue, masking was required and enforced. I always mask indoors (no exceptions), so I definitely felt more at ease among the droves of humans, knowing that we had those protections in place. The other thing that I very much appreciated is that the CNCF provided freebie rapid COVID tests proctored through eMed, so I was able to test daily while I was there, for extra peace of mind.

Although KubeCon itself didn’t start until Wednesday, October 26th, I flew in on Sunday night so that I could catch Open Observability Day on Monday, and OTel Unplugged on Tuesday.

Day 1: Open Observability Day

I popped in and out of Open Observability Day, and of the talks that I caught, there were two talks that I really enjoyed. One was OTel Me How to Build a Data Pipeline for Observability, by Daniel Kim and Reese Lee. Even though I was already familiar with the content, I really appreciated Daniel and Reese’s super energetic presentation style. It was definitely a welcome pick-me-up to help fight post-lunch food coma. The other talk I really enjoyed was What Can eBPF Actually do for Modern-Day Observability? by Ori Shussman. In it, he talks about how eBPF lets us see data that is otherwise not visible. For example, eBPF is useful for providing greater insight into gRPC calls, which are notoriously difficult to observe. 🤯I also wanted to give a special shout-out to Opening the Door to Observability, by Libby Meren. Although I missed this talk, this topic is very near and dear to my heart, because in my previous role, I had to spend a chunk of time trying to get buy-in on doing Observability The Right Way ™.

Interested in checking out the other talks? Y’all can check out videos for them here.

Later that afternoon, I got to meet my Lightstep DevRel fam in real life, and it was AWESOME! Unfortunately, we never took a group selfie. 😿

Day 2: OTel Unplugged

I spent Day 2 at the Colony Club to attend OTel Unplugged. This event was sponsored by Lightstep, Honeycomb, New Relic, Splunk, Dynatrace, Crowdstrike, and NGINX. I came into the event not knowing what to expect. I can sometimes clamp up when I’m around folks that I don’t know, but because I was helping with the event check-in, I got to say hello to a number of the attendees, which helped break the ice. And it turns out that there were a lot of names that I recognized from my work in the OTel community, and it was nice to connect in person with folks whom I’d only previously met through Slack or Zoom.

PS: Big shout-out to the venue, which was gorgeous, and the staff were super friendly.

Day 3: KubeCon!

Finally, the main event! I’d been to the conference center on Monday to pick up my badge, but the volume of people that day didn’t even compare to how busy things got on Wednesday. It was a whirlwind of a day, and I spent some of my time at the Lightstep booth. The Lightstep crew did demos of the OTel Demo App to illustrate Observability-Landscape-as-Code in action, which Ana and I poured our blood, terror, terror, sweat, and (happy?) tears into it before KubeCon. It’s a great little demo, if I do say so myself, and I definitely recommend that you check it out. (Shameless plug, I know. Sorrynotsorry.)

The life-sized game of Jenga at our booth was a huge draw, and there were a lot of adventurous folks who tried their hand at it. The pic of Ana and me below shows how tall the tower got. What you don’t see is that, shortly after the pic was taken, one of us knocked the tower down. Oops. 😳

Our team also had KubeCon-Detroit-themed stickers made especially for this event. Ana took things to the next level and did her nails to match our stickers. And I don’t know if you can see it in any of the photos below, but she also wore Kubernetes-themed earrings too. She’s very extra, and I love it! The stickers were part of a scavenger hunt that revealed the location of a pizza party hosted by Lightstep on Thursday night.

Ana and I also spent some time roaming the exhibitors’ area, grabbing some sweet swag (I nabbed a sweet OTel t-shirt and the cutest little Tracetest plushie, whom I named Tracey), meeting new and interesting people, and meeting some familiar faces in real life. (Hi Abby!!) Ah, the life of a DevRel. So. Much. Fun. 💜💜💜

Unfortunately, I wasn’t able to stay for the full conference, but from what I did experience, it was lots of fun. This most definitely will not be my last KubeCon, and I most definitely can’t wait for the next KubeCon! Bring it!

Ana’s Point-of-View

Ana here! I wanted to provide my perspective from attending four KubeCons in the past. This is my second KubeCon through COVID, and I appreciate that the CNCF has enforced COVID protocols for their events. As someone who caught COVID during KubeCon EU València, I was extra careful this time around, and tested daily. I’m still testing and COVID negative, so thanks, CNCF!

On Monday, I was scheduled to go to the Contributor Summit, but I lost my voice the week before on Thursday, and I still hadn’t recovered it by Monday morning.🥲 I had talks and booth duty during the coming week, so instead I did something more easygoing and I just helped my fellow Lightsteppers get ready for KubeCon and did final touches on my slides.

One thing that was really hard this year was that there were too many (or what felt to me like too many) co-located/Day 0 events. It’s great to see the ecosystem continuing to grow, but it definitely makes it very hard to decide which events to go to, for an opportunity to learn from the community. I know I struggled with this, so I do hope that we try to unify topics a bit more next time.

On Tuesday, I split my time between Chaos Day, Keptn Community Day, and of course, the hallway track of the co-located events. From working in the Chaos Engineering community the last 4+ years, I loved to see the Cloud Native Chaos Engineering community growth, I also loved talking to folks about the future of Keptn, and giving a talk on how to achieve more reliability with this CNCF sandbox project and the other OSS tools.

As Wednesday kicked off, it was great to feel the buzz and excitement of folks so eager to learn and connect. From Day 1’s keynote, I really enjoyed the reminder that companies benefiting from Kubernetes and the CNCF ecosystem should be getting involved, giving back, and mentoring others. There was a lot of love to maintainers and contributors in the ecosystem, and as someone involved in the Kubernetes Release Team for Kubernetes v1.25 and v1.26, it was a huge honor to see my face up in there with the rest of the team! 💙

I also took some time to walk the show floor, buy some CNCF swag and check out the vendor swag too.

Thursday was another busy day of running around with the community. I also got to record some Cloud Native in Spanish content with some other Latinxs in the space. In addition to that, Lightstep hosted two events, one of which was our secret pizza party. It was fun to take over Detroit’s famous alley, The Belt, to hang out and talk about SRE. Thanks to everyone who stopped by! #DetroitSRECity 🤘

One of the other places I spent a lot of time at at KubeCon NA was the CNCF Project Pavillion. I was very happy to see that it was a bit larger than the area we had during KubeCon EU, but I still wish that it was bigger and wasn’t so tucked away in the corner. A number of booths were showcasing their projects with demos through the week, hosted Q&A time, and gave away swag. If you are still trying to understand the Cloud Native Ecosystem, you can look at this very extensive map of the landscape and projects under the CNCF, some of which are more advanced than others. Of course I’m biased, but I’m really excited for the work that Keptn is doing in helping developers have more control over their application lifecycle. I’m also very excited to see where Backstage goes and how other CNCF projects can integrate with their service catalog.

On Friday, I got a chance to hang out some more with my community friends and give a talk on the future of Keptn with our friends at Dynatrace. I am very excited to see where this project goes and what work we continue to do with OpenTelemetry to make it easier to observe our deployments and their reliability.

Final Thoughts

Aaaand...that’s a wrap! KubeCon was a ton of fun for both Ana and me. We loved having the Lightstep DevRel team in person for the first time, we loved connecting with our various CNCF communities, Lightstep customers, and it was great fun to see all the cool tech out there. Whether you’re a KubeCon newbie like me, or KubeCon veteran like Ana, it was great to see the CNCF community continuing to come together, grow, and prosper. We are definitely looking forward to KubeCon EU in 2023!

Now, please enjoy this photo of Adriana’s rat, Phoebe, keeping her company as she does her KubeCon expense report.

Peace, love, and code. 🦄 🌈 💫

Got questions about Observability? Talk to us! Feel free to connect with us through e-mail, or:

Connect with Adriana up on Twitter or LinkedIn
Connect with Ana on Twitter or LinkedIn.

Hope to hear from y’all!

Running the OpenTelemetry Demo App in Kubernetes

Adriana Villela — Fri, 28 Oct 2022 15:30:00 +0000

If you’re new to OpenTelemetry and want to see it in action in a real-life example, the OpenTelemetry Demo app is a great way to get started quickly. In one of my previous blog posts, I showed you how to get the demo app up and running and sending Traces to Lightstep. That’s all well and good, but we all know that in “The Real World”, we’re not running our containerized workloads locally with Docker Compose. Instead, we’re running them using container orchestrators such as Kubernetes and Nomad. Keeping that in mind, wouldn’t it be so very very nice if we could run the OpenTelemetry Demo app in Kubernetes? And wouldn’t it be super extra nice if we could send those Traces to Lightstep?

Well, look no further, my friend, because today, that is exactly what we’re going to do!

Are you ready? Let’s do this!

Tutorial

Pre-Requisites

Before you begin, you will need the following:

A Lightstep account
A Lightstep Access Token for the Lightstep project you would like to use
A Kubernetes cluster
Helm

NOTE: The version of the OpenTelemetry Demo App used at the time of this writing was version 0.9.6.

Steps

1- Initialize Helm

helm repo add open-telemetry https://github.com/open-telemetry/opentelemetry-helm-charts

2- Create the values YAML file

The OTel Demo App Helm Chart expects a values.yaml file. Now, the one that comes out-of-the-box with the chart is all well and good, but it sends Traces to Jaeger. In our case, since we want to send Traces to Lightstep, we’ll need to modify it a tad. Let’s start by creating our own values YAML file:

touch values-ls.yaml

And then open values-ls.yaml and add:

mkdir otel-demo-ls
tee -a values-ls.yaml <<EOF
opentelemetry-collector:
  nameOverride: otelcol
  mode: deployment
  extraEnvs:
    - name: LS_TOKEN
      valueFrom:
        secretKeyRef:
          key: LS_TOKEN
          name: otel-collector-secret
  resources:
    limits:
      memory: 100Mi
  config:
    exporters:
      otlp/ls:
        endpoint: ingest.lightstep.com:443
        timeout: 30s
        tls:
          insecure_skip_verify: true
        headers:
          "lightstep-access-token": "${LS_TOKEN}"

      otlp:
        endpoint: '{{ .Release.Name }}-jaeger:4317'
        tls:
          insecure: true

      logging:
        logLevel: debug

    processors:
      resource:
        attributes:
        - key: service
          from_attribute: service.name
          action: insert
      resource/add_container:
        attributes:
        - key: container
          from_attribute: service.name
          action: insert

    service:
      pipelines:
        metrics:
          exporters:
            - logging
            - otlp/ls
          processors:
            - resource
        traces:
          exporters:
            - logging
            - otlp
            - otlp/ls
          processors:
            - resource/add_container
EOF

NOTE: I had originally left out Jaeger and Prometheus configurations for the sake of keeping things simple. If you'd like to see an example of with both, check out this here. (Thank you Vamsee Lakamsani for the share!)

Your file should look like this one. Okay...so what exactly did we do?? So glad you asked!

Well, in a nutshell, we’re overriding the values.yaml in the OpenTelemetry Demo App Chart. But we’re only overriding the bits that we need to override - specifically, the OTel Collector configuration, so we can send Traces to Lightstep. It looks an awful lot like the Collector config YAML that we know and love. But not quite. Because it’s just a partial config of the Collector config. You see, Demo App Chart uses the OpenTelemetry Collector Helm Chart as a subchart. All the configuration that the Collector Chart exposes is available to us in the Demo Chart. So when you’re populating your own version of values.yaml, all you need to do is include the Collector configs that you wish to modify or add to. In our case, we’re doing the following:

Configure a new exporter, otlp/ls, which allows us to send traces to Lightstep
Add the new exporter to our metrics and traces pipelines
Update the logging exporter to use the debug log level.

You’ll notice that in configuring the otlp/ls exporter, we’re setting the following header value: "lightstep-access-token": "${LS_TOKEN}". But where in Space does ${LS_TOKEN} come from?? Great question! Which brings me to the second noteworthy section.

You may have noticed the extraEnvs section in our opentelemetry-collector config. Well, this is where we can configure environment variables that are mounted to our Collector pod in Kubernetes. We don’t wish to expose our secret in values-ls.yaml, as that would be a security no-no. Instead, we reference a secret called otel-collector-secret, which is mounted as the environment variable, LS_TOKEN. Ta-da! 🎉

PS: We’ll create the secret in the next step.

NOTE: I am fully aware of the fact that you won’t want to use Kubernetes secrets In Real Life to store your Lightstep Access Token, as they are only base64-encoded. Instead, you’ll want to store your secrets in a secrets manager, such as one that comes with your Cloud provider (e.g. Azure Key Vault, Google Secret Manager), or HashiCorp Vault.

3- Deploy the app

Now that we know what’s up, let’s deploy the app to Kubernetes!

export LS_TOKEN="<YOUR_LS_TOKEN>"

kubectl create ns otel-demo
kubectl create secret generic otel-collector-secret -n otel-demo --from-literal=LS_TOKEN=$LS_TOKEN

helm upgrade my-otel-demo open-telemetry/opentelemetry-demo -f <path-to-values-ls-file>/values-ls.yaml -n otel-demo --install

Where <path-to-values-ls-file> is the path in which your newly-created values-ls.yaml is located.

Be sure to replace <YOUR_LS_TOKEN> with your own Lightstep Access Token.

4- Access the OTel Demo App

You can access the Demo App by Kubernetes port-forward:

kubectl port-forward -n otel-demo svc/otel-demo-app-frontend 8080:8080

To access the front-end, go to http://localhost:8080:

5- See traces in Lightstep

We can now pop over to Lightstep and check out some Traces. Let's do this by creating a Notebook.

First, click on the little page icon on the left nav bar (highlighted in blue, below). That will bring up this page:

Next, we build our query for our Traces. Let's look at the traces from the recommendationservice. We'll do by entering recommendationservice in the field next to "All telemetry". Because this is a service, select the second value from the drop-down, which says, "Use 'recommendationservice' as service value", as per below:

After you select that value, you'll see a chart like this:

The little green dots represent trace exemplars from that Service. Hover over one of them to see for yourself!

If you click on one of these dots, you'll get taken to the Trace view. Before you click, be sure to save your Notebook first (don't worry, you'll get a reminder before you navigate away from the page)!

Here's the Trace view we see when we click on the get_product_list dot (Operation) above:

Final Thoughts

Today we upped our OpenTelemetry Demo App game, and moved from running it à la Docker Compose, to deploying it to Kubernetes. We did this, thanks to the OpenTelemetry Demo App Helm Chart, and we used our own version of values.yaml so that we could send Traces and Metrics to Lightstep.

This should give you a nice feel for running a full-fledged OTel-instrumented app in Kubernetes!

And now, I shall reward you with a picture of Ollie Octopus painting, drawn by my superly-talented 14-year-old daughter.

Peace, love, and code. 🦄 🌈 💫

The OpenTelemetry Demo App is always looking for feedback and contributors. Consider joining the OTel Community to help make OpenTelemetry AWESOME!

Got questions about today’s blog post? Talk to me! Feel free to connect through e-mail, Twitter or LinkedIn.

Hope to hear from y’all!

Forem: Adriana Villela

The Humans of OpenTelemetry: KubeCon EU 2024

The Humans of OTel - KubeCon EU 2024 - YouTube

Transcript

1- Meet the Humans of OTel

2- What does observability mean to you?

3- What does OpenTelemetry mean to you?

4- How did you get involved with OpenTelemetry?

5- What’s your favorite telemetry signal?

Join us!

Demystifying Observability 2.0

Defining Observability

Observability 1.0

1- Yow you operate your code

2- Known unknowns

3- Multiple sources of truth

Observability 2.0

1- It’s focused not only on how you operate your code, but also on how you develop your code

2- It’s focused on unknown unknowns

3- It’s focused on a single source of truth: events

Final thoughts

Things You Might Not Have Known About the OpenTelemetry Operator

Q&A

Q1: Does the Operator support multiple Collector configuration sources?

Q2: How can I securely reference access tokens in the Collector’s configuration?

Q3: Is the Operator version at parity with the Collector version?

Q4: Can I override the base OTel Collector image?

Q5: Does the Target Allocator work for all deployment types?

Q6: If I’m using Operator’s Target Allocator for Prometheus service discovery, do I need PodMonitor and ServiceMonitor CRs installed in my Kubernetes cluster?

Q7: Do I need to create a service account to use the Target Allocator?

Q8: Can I override the Target Allocator base image?

Q9: If it’s not recommended that you override the Target Allocator base image, then why would you want to?

Q10: Is there a version lag between the OTel Operator auto-instrumentation and auto-instrumentation of supported languages?

Final Thoughts

OpenTelemetry Collector Anti-Patterns

Antipatterns

1- Improper use of Collector deployment modes

2- Not monitoring your Collectors

3- Not using the right Collector Distribution (or not building your own distribution)

4- Not updating your Collectors

5- Not using the OpenTelemetry Collector where appropriate

Final Thoughts

Tracetest in Action: Running Trace-Based Tests on the OpenTelemetry Demo App with Nomad

Tracetest Overview

Tutorial

Assumptions

Pre-Requisites

Tutorial Repos

Part 1: Provision Nomad, Consul, and Vault using HashiQube

1- Update /etc/hosts

2- Provision a Local Hashi Environment with HashiQube

Part 2: Deploying Tracetest and the OTel Demo App on Nomad

1- Add Lightstep Access Token to Vault

2- Deploy the OTel Demo App and Tracetest

3- See it in Nomad

4- Access the OTel Demo App

5- Access Tracetest

Part 3: Creating & Running Tests in Tracetest

1- Install the Tracetest CLI

2- Configure the Tracetest CLI

3- Create a Trace-based test in Tracetest

4- Run the Trace-based test

Part 4: See it Lightstep!

Final Thoughts

Running Tracetest and the OpenTelemetry Demo App on HashiCorp Nomad

How to Convert Kubernetes Manifests into Nomad Jobspecs

Manifests and Helm Charts and Jobspecs…oh my!

Conversion Process

1- Grab the Kubernetes manifests

2- Prepare the jobspec

3- Port definitions

4- Service Definition

5- Task Definition

Config Stanza

Env Stanza

Template Stanza

Restart Rules

Resource Allocations

6- The Final Product!

Final Thoughts

Q6: If I’m using Operator’s Target Allocator for Prometheus service discovery, do I need `PodMonitor` and `ServiceMonitor` CRs installed in my Kubernetes cluster?