Forem: Aditi Ahuja

Thanos: Musings of a Mentee

Aditi Ahuja — Sun, 19 Dec 2021 06:01:10 +0000

This is my account of my experience as a Fall ‘21 LFX mentee working with the Thanos project. The mentorship, as an experience, deserved an exhaustive post in its own right and this is my attempt at penning down the events this fall, in as much detail as I could.

This post is divided into 4 sections:

Project
The Thanos Process
Getting started as a contributor
Learnings

Project
My project involved adding a feature flag to enable Prometheus metrics which tracked the progress of compaction, downsampling and retention processes. These metrics are useful when performing a time-consuming compaction on large blocks of TSDB data. Prior to the project, compaction progress would have to be manually monitored using the bucket UI.The added metrics export data that can be visualised graphically and would give an estimate about some progress parameters related to these processes.I also added unit tests and the required documentation for the functions added as part of the project.

Some other additions I worked on include adding tracing support for compaction and adding the enable-feature flag for negative offsets and @ modifiers in Thanos compact.

For understanding my work better, these are some links:

The PR for compaction and downsampling progress metrics
a. Diagram to understand compaction metrics.
The PR for retention metrics
The PR for compaction tracing support

The Thanos Process
Since my mentor and I were in timezones 11 and a half hours apart, our weekly sync up was on Sunday morning(for me)/Saturday night(for him). Time zones notwithstanding, we frequently communicated synchronously.

The first few meetings involved clarifying the project goals and understanding the Thanos compactor. Each meeting had a defined agenda and actionables for the next week.

Most meetings had me relentlessly asking my mentor all kinds of queries (Some in hindsight, very trivial). Each meeting would end with my mentor detailing the week’s actionables to me and each week would be peppered with queries on the nuts and bolts of the implementation. When I started on my first major PR(which was ultimately not merged), I was completely off the mark when coding my first draft. My mentor had to then break down the implementation into steps to get me onto the right track.

My project PRs were frequently reviewed by my mentor and I tried to resolve the comments as fast as possible. This led to PRs with a short review-response cycle with some intense discussions and a very steep learning curve from the reviews.

Getting started as a contributor
The first step to familiarising myself was setting up Thanos locally. In the process, I discovered that an interactive example lacked instructions and opened a PR for the same. This was my first PR and undoubtedly an easy one, which boosted my confidence a teeny bit.

The mentorship was the first time I was contributing to Thanos. To get acquainted with the code, my mentor suggested I take on some good first issues. My first such issue involved refactoring a file and had earlier PRs refactoring other files similarly, for reference. Even though this PR was relatively straightforward in terms of coding(since I wasn’t completely new to Go), it taught me about the Thanos PR process, which, incidentally, is very thorough. The reviewers found nits as small as an extra pair of parentheses in my commits!

Taking the initiative to work on small issues definitely paved the way for a smoother mentorship when I was working on my project. It helped me in many ways, the most obvious one being getting familiarised with navigating the codebase and the review process. In hindsight, I realise that one of the other ways it benefited me was the confidence I developed when I took on an issue, however trivial, discussed the approach with a maintainer and then iterated based on the reviews. The supportive community definitely merits a mention here and had it not been for the encouragement extended to the mentees, this would have been an intimidating task.

Learnings
Prior to the mentorship, I thought I was experienced enough in Go, having coded in it for roughly a year. What did I know, that myth would be soon busted! One of the things that caught my eye as a new contributor was the style guide. Opinionated and quite exhaustive, the style guide has some really interesting Go conventions. The few that struck with me are:

Pre-allocating slices and maps if we are aware of their size
Using values of type struct{} with maps since they are allocated with 0 bytes
Handling errors from deferred functions.

Since my project involved adding metrics to a Prometheus exporter, I learnt about the types of metrics and developed a small side project to export some data. This was interesting and came in useful when I had to add metrics related to compaction, downsampling and retention to the upstream codebase.

Initially, I struggled to understand how I would pare down the compaction process to its essentials to simulate it and predict the blocks and compaction runs. My mentor then had to explain it to me using skeletal pseudo code, which cleared the fog a little but the implementations of some concepts like filter and syncer were still hazy. I understood those better in the last phase when after asking my nth trivial doubt, my mentor explained a crucial fact about syncer’s working which, in hindsight, explained the implementation details of code I had already worked on for a prior feature. The next phase, working on downsampling simulation, saw me a tiny bit more independent since the codebase is relatively more straightforward compared to the compaction codebase. I still considered adding some non-essentials to the process, which my mentor had to tell me were unnecessary. The last leg of the project involved working on a stretch goal, adding retention metrics. This was fairly straightforward since the retention codebase seemed more intuitive and it was the third time I was adding metrics for a process. Third time’s the charm :)

The exhaustive review process(easily a 100+ review comments, led by my indefatigable mentor) taught me a lot about considering some of the finer aspects of a feature, usually related to Prometheus and the Go SDK. These ranged from always specifying a registry when creating a new metric with promauto to avoid registering with the default global registry, using Reset() to clear earlier exported values and the basics of promtestutil for related testing.

Another area I learnt a lot about was structuring my code including,surprisingly, naming functions/variables and writing comments!
The first draft of my PR lacked the ‘code consistency’ of the final PR in terms of proper use of interfaces, consistent function signatures and abstraction of related data structures. Aside: the Thanos codebase is a pretty good example of a well structured Go codebase. Definitely worth your while exploring it!
I could never get variable/struct/function names right and fixing them seemed like a never-ending process during the reviews. Turns out, I either did not comment enough or when I did, my comments were too detailed and not abstract enough. Always remember: All comments at Thanos end with a period. Period.

The first major PR, which I worked hard on for a couple of weeks,was made redundant. It not being merged did dampen my morale for a tiny bit. However, apart from the technical learnings about the internals of compactor, the overarching lesson was not letting my enthusiasm dampen when faced with such a situation and not considering the time spent as a sunk cost. This, I would say, was one of the most important lessons from the mentorship.

To sum it up, the mentorship gave my progression as a developer a fillip in terms of not just technical and non-technical knowledge, but fostered a connection with an inspiring, supportive community and made me aware of interesting opportunities and perspectives which I wouldn’t have been if not for the mentorship. Truly thankful to everyone who helped me in this journey, my mentor, my co-mentees, the Thanos maintainers and the wider Thanos community!

Custom Prometheus Metrics with Go

Aditi Ahuja — Sun, 26 Sep 2021 12:58:06 +0000

This post is intended to be a guide for understanding and working on Prometheus instrumentation using the go client. I personally did not feel any existing content on this was comprehensive enough and hence, decided to pen down my lessons from recent trial-and-error development.

End Goal

Developing an exporter for weather data fetched using the API.

All of the code referred to in the post is from this repo.

Prerequisites

A working installation of Prometheus running on port 9090
Go 1.15+
An API key from tomorrow.io
A basic understanding of metrics and exporters - this is an article I recommend. I will be covering the practical aspects related to developing and using them.

Querying the weather API

Our GET request will query the API to gather data about the temperature in metric units at specific coordinates, with a 1 hour interval between data points.

The GET request can be made using NewRequest from net/http. Replace the "APIKEY" string with your API key.

url := fmt.Sprintf("https://api.tomorrow.io/v4/timelines?location=%f,%f&fields=temperature&timesteps=%s&units=%s", 73.98529171943665, 40.75872069597532, "1h", "metric")

req, err := http.NewRequest("GET", url, nil)
if err != nil {
        return Response{}, errors.New("error in GET request")
}
req.Header.Add("apikey", "APIKEY")
res, err := http.DefaultClient.Do(req)
if err != nil {
        return Response{}, err
}
defer res.Body.Close()

Since the response data is structured JSON, we create structs with specific fields to unmarshal the response. This is done using the encoding/json package.

For the struct fields, look at this part of the repo. In the interests of brevity, I will not be adding those snippets here.

body, err := ioutil.ReadAll(res.Body)
if err != nil {
        return Response{}, errors.New("error reading response")
}

var dat Response
if err := json.Unmarshal(body, &dat); err != nil {
        return Response{}, errors.New("error unmarshalling JSON")
}

Developing a custom exporter

I am going to implement an exporter two ways, asynchronous and synchronous.

Jobs and Registries

Many target endpoints with the same purpose combine to form a job.

Serving Metrics

Metrics can be 'served' i.e. exposed at a port of your choice. Here, I have chosen port 2112. I have chosen '/metrics' as my endpoint.

Asynchronous

In this type of exporter, metrics are collected asynchronously. The function to collect the metrics is run asynchronously.

go recordMetrics()

The two metrics we will implement are gauges, opsProcessed and tempCelsius

Here is the initialisation for tempCelsius:

var tempCelsius = promauto.NewGauge(
    prometheus.GaugeOpts{
        Name: "current_temperature_api_celsius",
        Help: "Current temperature",
    },
)

Name should be a fully qualified Prometheus metric name and Help is the description that appears on hovering over the metric in the UI.

Since we are using promauto to initialise them, we do not need to explicitly add these to the registry.

Each call to the API returns multiple temperatures, each of which will be treated as a separate data point. To add these to the gauge, we will use gauge.Set(dataPoint) as shown:

for _, interval := range dat.Data.Timestep[0].TempVal {
            tempCelsius.Set(interval.Values.Temp)
}

The above code snippet is an example of 'direct instrumentation' where the same metrics are updated on each scrape.

Synchronous. We first create a registry and register the standard Process and Go metrics.

reg := prometheus.NewRegistry()
reg.MustRegister(
        collectors.NewProcessCollector(collectors.ProcessCollectorOpts{}),
        collectors.NewGoCollector(),
)

The metrics we will be monitoring are temperature fetched using the weather API, the time taken for those requests and the humidity percentage.

This manner of metric exporting involves 'collectors'. A collector simply represents a set of metrics. In contrast to 'direct instrumentation', this involves creating new metrics each time with MustNewConstMetric.

Let's first describe the temperature and humidity metrics. This is the description for temperature:

tempDesc = prometheus.NewDesc(
        "temperature_city_fahrenheit",
        "temperature of a city in fahrenheit",
        []string{"city"}, nil,
)

The first parameter is the metric name, this is the name that will be used when querying the metric from the UI.

The second is the description/help string. It gives a brief description of the metric.

A struct CityStats is defined. Based on this struct, the Collect and NewCityStats functions, and the CityStatsCollector interface are defined.

func (cc CityStatsCollector) Collect(ch chan<- prometheus.Metric) {
    tempByCity, humidityByCity := cc.CityStats.TemperatureAndHumidity()

    for city, temp := range tempByCity {
        ch <- prometheus.MustNewConstMetric(
            tempDesc,
            prometheus.CounterValue,
            float64(temp),
            city,
        )
    }
}

Here, the TemperatureAndHumidity function basically returns the results of the API call. Each of those are added to the metric channel ch using MustNewConstMetric.

Setup

Let's understand the structure of the Prometheus config file used with such exporters.

We will be working with a very basic version of the file.

First, we have the global rules. Here, we will focus on 'scrape_interval'.

Scrape interval refers to the time period between two consecutive scraping runs. Scraping is the process of gathering data for all the metrics from specific endpoints or 'targets'.

global:
  scrape_interval: 15s

Next up, we have the scrape configs. Each job here should be assigned a unique 'job_name'.

Job configs can be specified in two ways, either statically or dynamically using the available service discovery mechanism.

For now, we will be focusing on the static configs.

- job_name: myapp
    static_configs:
      - targets:
        - localhost:2112

The endpoints we mentioned above, 'targets' are assigned to a job and are one or more monitored URL endpoints that Prometheus will scrape metrics from.

Run Prometheus as usual with the config file specs discussed above.
To run either the synchronous or asynchronous exporters, just run them using go run sync.go temp.go or go run async.go temp.go.

Hopefully this was a good primer on getting started with exporting custom metrics. Would love to discuss and hear feedback in the comments.