Forem: Akash Warkhade

Achieve Multi-tenancy in Monitoring with Prometheus & Thanos Receiver

Akash Warkhade — Sat, 24 Jun 2023 14:39:59 +0000

Hey there! If you are reading this blog post, then I guess you are
already aware of Prometheus and how it helps
us in monitoring distributed systems like Kubernetes. And if you are familiar with
Prometheus, then chances are that you have come across the tool called
Thanos. Thanos is a popular OSS that helps
enterprises achieve a HA Prometheus setup with long-term storage
capabilities. One of the common challenges of distributed monitoring is
to implement multi-tenancy. Thanos
receiver is a Thanos
component designed to address this common challenge.
Receiver was part of Thanos for a significant duration as an experimental feature. However, after some time, it reached the general availability stage and is now fully supported.

A few words on Thanos Receiver

Receiver is a Thanos component that can accept remote
write
requests from any Prometheus instance and store the data in its local
TSDB, optionally it can upload those TSDB blocks to an object
storage like S3 or GCS at
regular intervals. Receiver does this by implementing the Prometheus
Remote Write
API.
It builds on top of existing Prometheus TSDB and retains their
usefulness while extending their functionality with long-term-storage,
horizontal scalability, and down-sampling. It exposes the
StoreAPI
so that Thanos Queriers
can query received metrics in real-time.

Multi-tenancy in Thanos Receiver

Thanos receiver supports multi-tenancy. It accepts Prometheus remote
write requests, and writes these into a local instance of Prometheus
TSDB. The value of the HTTP header (“THANOS-TENANT”) of the incoming
request determines the id of the tenant Prometheus. To prevent data
leaking at the database level, each tenant has an individual TSDB
instance, meaning a single Thanos receiver may manage multiple TSDB
instances. Once the data is successfully committed to the tenant’s TSDB,
the requests return successfully. Thanos Receiver also supports
multi-tenancy by exposing labels which are similar to Prometheus
external
labels.

Hashring configuration file

If we want features like load-balancing and data replication, we can run
multiple instances of Thanos receiver as a part of a single hashring.
The receiver instances within the same hashring become aware of their
peers through a hashring configuration file. Following is an example of
a hashring configuration file.

[
   {
       "hashring": "tenant-a",
       "endpoints": ["tenant-a-1.metrics.local:19291/api/v1/receive", "tenant-a-2.metrics.local:19291/api/v1/receive"],
       "tenants": ["tenant-a"]
   },
   {
       "hashring": "tenants-b-c",
       "endpoints": ["tenant-b-c-1.metrics.local:19291/api/v1/receive", "tenant-b-c-2.metrics.local:19291/api/v1/receive"],
       "tenants": ["tenant-b", "tenant-c"]
   },
   {
       "hashring": "soft-tenants",
       "endpoints": ["http://soft-tenants-1.metrics.local:19291/api/v1/receive"]
   }
]

Soft tenancy – If a hashring specifies no explicit tenants, then any tenant is considered a valid match; this allows for a cluster to provide soft-tenancy. Requests whose tenant ID matches no other hashring explicitly, will automatically land in this soft tenancy hashring. All incoming remote write requests which don’t set the tenant header in the HTTP request, fall under soft tenancy and default tenant ID (configurable through the flag –receive.default-tenant-id) is attached to their metrics.
Hard tenancy – Hard tenants must set the tenant header in every HTTP request for remote write. Hard tenants in the Thanos receiver are configured in a hashring config file. Changes to this configuration must be orchestrated by a configuration management tool. When a remote write request is received by a Thanos receiver, it goes through the list of configured hard tenants. A hard tenant also has the number of associated receiver endpoints belonging to it.

P.S: A remote write request can be initially received by any receiver
instance, however, will only be dispatched to receiver endpoints that
correspond to that hard tenant.

Architecture

In this blog post, we are trying to implement the following
architecture. We will use Thanos v0.31.0 in this blog post.

A simple multi-tenancy model with Thanos Receiver

Brief overview on the above architecture:

We have 3 Prometheuses running in namespaces: sre, tenant-a and
tenant-b respectively.
The Prometheus in sre namespace is demonstrated as a soft-tenant
therefore it does not set any additional HTTP headers to the remote
write requests.
The Prometheuses in tenant-a and tenant-b are demonstrated as
hard tenants. The NGINX servers in those respective namespaces are
used for setting tenant header for the tenant Prometheus.
From security point of view we are only exposing the Thanos receiver
statefulset responsible for the soft-tenant (sre Prometheus).
For both Thanos receiver statefulsets (soft and hard) we are setting
a replication
factor=2.
This would ensure that the incoming data get replicated between two
receiver pods.
The remote write request which is received by the soft tenant
receiver
instance is forwarded to the hard tenant thanos
receiver.
This routing is based on the hashring config.

The above architecture obviously misses few features that one would also
expect from a multi-tenant architecture, e.g: tenant isolation,
authentication, etc. This blog post only focuses how we can use the
Thanos Receiver to store time-series from multiple Prometheus(es) to
achieve multi-tenancy. Also the idea behind this setup is to show how we
can make the prometheus on the tenant side nearly stateless yet maintain
data resiliency.

We will improve this architecture, in the upcoming posts. So, stay
tuned.

Prerequisites

KIND / managed cluster / minikube (We will be using Kind)
kubectl
helm
jq(optional)

Cluster setup

Clone the repo:

 git clone https://github.com/infracloudio/thanos-receiver-demo.git

Setup a local KIND cluster

Move to the local-cluster directory:
```
cd local-cluster/
```
Create the cluster with calico, ingress and extra-port mappings:
```
./create-cluster.sh cluster-1 kind-calico-cluster-1.yaml
```

Deploy the nginx ingress controller:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install nginx-controller ingress-nginx/ingress-nginx -n ingress-nginx --create-namespace

Now, move back to the root directory of the repo:
```
cd -
```

Install minio as object storage

helm repo add bitnami https://charts.bitnami.com/bitnami

Install minio in the cluster:

helm upgrade --install my-minio bitnami/minio \
  --set ingress.enabled=true --set auth.rootUser=minio --set auth.rootPassword=minio123 \
  --namespace minio --create-namespace

    kubectl port-forward svc/my-minio 9001:9001 -n minio
    ```
{% endraw %}

4.  if you face  **E0528 13:02:43.145873   44832 portforward.go:346] error creating error stream for port 9001 -> 9001: Timeout occurred** issue while doing port-forward for minio then execute above port-forward command in one terminal and then open another terminal and execute below command which will keep connection to minio pods keep running.
{% raw %}

    ```sh
    while true ; do wget 127.0.0.1:9001 ; sleep 10 ; done
    ```
{% endraw %}

5.  then Login to minio by opening http://localhost:9001/ in browser with credentials username minio and password minio123
6.  Create a bucket with name **thanos** from UI

### Install Thanos components

**Create shared components**
{% raw %}


```shell
kubectl create ns thanos

## Create a file _thanos-s3.yaml_ containing the minio object storage config for tenant-a:
cat << EOF > thanos-s3.yaml
type: S3
config:
  bucket: "thanos"
  endpoint: "my-minio.minio.svc.cluster.local:9000"
  access_key: "minio"
  secret_key: "minio123"
  insecure: true
EOF

## Create secret from the file created above to be used with the thanos components e.g store, receiver
kubectl -n thanos create secret generic thanos-objectstorage --from-file=thanos-s3.yaml
kubectl -n thanos label secrets thanos-objectstorage part-of=thanos

## go to manifests directory
cd ../manifests/

Install Thanos Receive Controller

Deploy a thanos-receiver-controller to auto-update the hashring
configmap when the thanos receiver statefulset scales:
```
kubectl apply -f thanos-receiver-hashring-configmap-base.yaml
kubectl apply -f thanos-receive-controller.yaml
```
The deployment above would generate a new configmap
thanos-receive-generated and keep it updated with a list of
endpoints when a statefulset with label:
controller.receive.thanos.io/hashring=hashring-0 and/or
controller.receive.thanos.io/hashring=default get created or
updated. The thanos receiver pods would load the
thanos-receive-generatedconfigmaps in them.

**NOTE: The **default* and hashring-0 hashrings would be
responsible for the soft-tenancy and hard-tenancy respectively.*

Install Thanos Receiver

Create the thanos-receiver statefulsets and headless services for
soft and hard tenants.

We are not using persistent volumes just for this demo.
```
kubectl apply -f thanos-receive-default.yaml 
kubectl apply -f thanos-receive-hashring-0.yaml
```
The receiver pods are configured to store 15d of data and with replication factor of 2
Create a service in front of the thanos receiver statefulset for the
soft tenants.
```
kubectl apply -f thanos-receive-service.yaml
```
The pods of **thanos-receive-default* statefulset would
load-balance the incoming requests to other receiver pods based on
the hashring config maintained by the thanos receiver controller.*

Install Thanos Store

Create a thanos store statefulsets.

  kubectl apply -f thanos-store-shard-0.yaml

We have configured it such that the thanos querier fans out queries to the store only for data older than 2w. Data earlier than 15d are to be provided by the receiver pods. P.S: There is a overlap of 1d between the two time windows is intentional for data-resiliency.

Install Thanos Querier

Create a thanos querier deployment, expose it through service and ingress.

  kubectl apply -f thanos-query.yaml

We configure the thanos query to connect to receiver(s) and store(s) for fanning out queries.

Install Prometheus(es)

Create shared resource

kubectl create ns sre
kubectl create ns tenant-a
kubectl create ns tenant-b

Install kube-prometheus-stack

We install the
kube-prometheus-stack

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm upgrade --namespace sre --debug --install cluster-monitor prometheus-community/kube-prometheus-stack \
  --set prometheus.ingress.enabled=true \
  --set prometheus.ingress.hosts[0]="cluster.prometheus.local" \
  --set prometheus.prometheusSpec.remoteWrite[0].url="http://thanos-receive.thanos.svc.cluster.local:19291/api/v1/receive" \
  --set alertmanager.ingress.enabled=true \
  --set alertmanager.ingress.hosts[0]="cluster.alertmanager.local" \
  --set grafana.ingress.enabled=true \
  --set grafana.ingress.hosts[0]="grafana.local"

Install Prometheus and ServiceMonitor for tenant-a

In tenant-a namespace:

Deploy a nginx proxy to forward the requests from prometheus to thanos-receive service in thanos namespace. It also sets the tenant header of the outgoing request
```
  kubectl apply -f nginx-proxy-a.yaml
```
Create a
prometheus and a servicemonitor to monitor itself
```
  kubectl apply -f prometheus-tenant-a.yaml
```

Install Prometheus and ServiceMonitor for tenant-b

In tenant-b namespace:

Deploy a nginx proxy to forward the requests from prometheus to
thanos-receive service in thanos namespace. It also sets the
tenant header of the outgoing request
```
  kubectl apply -f nginx-proxy-b.yaml
```
Create a prometheus and a servicemonitor to monitor itself
```
  kubectl apply -f prometheus-tenant-b.yaml
```

Test the setup

Access the thanos querier by port-forwarding thanos-query service

kubectl port-forward svc/thanos-query 9090:9090 -n thanos

and open the thanos query UI by opening http://localhost:9090/ in the browser,
execute the query count(up) by (tenant_id).

Otherwise, if we have jq
installed, you can run the following command:

curl -s http://localhost:9090/api/v1/query?query="count(up)by("tenant_id")"|jq -r '.data.result[]|"\(.metric) \(.value[1])"'
{"tenant_id":"a"} 1
{"tenant_id":"b"} 1
{"tenant_id":"cluster"} 17

Either of the above outputs show that, cluster, a and b prometheus
tenants are respectively having 17, 1 and 1 scrape targets up and
running. All these data are getting stored in thanos-receiver in real
time by prometheus’ remote write
queue.
This model creates an opportunity for the tenant side prometheus to be
nearly stateless yet maintain data resiliency.

In our next post, we would improve this architecture to enforce tenant
isolation on thanos-querier side.

I hope you found this blog informative and engaging. If you encounter

References:

Testing your Infrastructure as Code using Terratest

Akash Warkhade — Wed, 15 Jun 2022 10:49:34 +0000

Setting Up infrastructure manually can be a time-consuming and hectic process. That is when we can make use of Infrastructure as Code (IaC) tools to automate the infrastructure. IaC automation can be done for any kind of Infrastructure i.e virtual machine, storage, etc. As more and more infrastructure becomes code, it is essential to have unit and integration tests for your IaC. We will briefly discuss what is IaC and testing your Infrastructure code mean. Then we deep dive into how we can use Terratest for IaC testing.

Let’s begin, shall we?

Infrastructure as code (IaC)

Infrastructure as Code is the process of provisioning and configuring an environment through code instead of manually setting up the required infrastructure and supporting system for it through GUI. For example, provisioning a virtual machine, configuring it, and setting up monitoring for it. Some of the IaC examples are Terraform, Packer, Ansible, etc. With the help of infrastructure as code, you can also track your infrastructure into a version control system such as Git, modularize and templatize in order to reuse the same code for multiple environments, and regions. Disaster recovery is one of the important benefits you get from coding your infrastructure. With IaC, you can replicate your Infrastructure in other regions or environments as quickly as possible.

Testing infrastructure code

IaC testing can be divided into multiple stages:

Unit testing
Integration testing
Sanity or Static Analysis

Sanity or Static Analysis

This is the very initial phase of testing your infrastructure code. In static analysis, we ensure that we have the correct syntax for our code. It also helps to ensure that our code is as per the industry standards and follows the best practices. Linters fall into this category. Some examples of sanity testing tools are foodcritic for Chef, hadolint for Docker, tflint for Terraform, etc.

Unit Testing

With the help of unit testing, we assess our code without actually provisioning the infrastructure. Examples can be restricting your container to run as a non-root user, or your cloud network security group should only have TCP protocols. Some of the unit testing examples are Conftest for Terraform, Chefspecs for Chef Cookbooks.

Conftest example for executing as a non root user:

package main

deny[msg] {
  input.kind == "Deployment"
  not input.spec.template.spec.securityContext.runAsNonRoot

  msg := "Containers must not run as root"
}

Integration testing

In integration testing, we want to test our IaC by actually deploying it into the required environment. For example, you deployed a virtual machine and hosted an Nginx server on port 80 on that machine. So you will check if port 80 is listening after deployment.

Below is the example of doing that with ServerSpec:

describe port(80) do
  it { should be_listening }
end

In this post, we are exploring integration testing of infrastructure code using Terrratest.

What is Terratest? What can we achieve with it?

Terratest is a Go library developed by Gruntwork that helps you create and automate tests for your Infra as Code written with Terraform, Packer for IaaS providers like Amazon, Google, or for a Kubernetes cluster. It provides you with various functions and patterns for tasks such as:

Testing Docker images, Helm charts, Packer templates.
Allows to work with various cloud provider APIs such as AWS, Azure.
Terratest executes sanity and functional testing for your Infrastructure code. With Terratest you can easily identify issues in your current infrastructure code and fix the issue as soon as possible. We can also leverage Terratest for compliance testing of your infrastructure, for example, to have versioning and encryption enabled on any new S3 bucket created through your IaC.

Installation of required binaries for Terratest

Terratest mainly requires Terraform and Go for execution. In this blog post, we have used Terraform version 1.0.0 and Go version 1.17.6 for our testing.

Installing Terraform

Follow the downloads section from Terraform website at to install Terraform on your machine, you can use package manager or download the binary and make it available in PATH.

After installation verify if it is installed properly by running the below command:

terraform version
Go & test dependency installation can be done with the following steps:

Installing Go

You can use your Linux distribution’s package manager to install Go, or follow the installation documentation of Go at.

Go test requires gcc for test execution
The go test command might require gcc, you can install it using your distribution’s package manager. For example, on CentOS/Amazon Linux 2, you can use yum install -y gcc.

Terratest in action

Now we will execute some integration tests using terratest. Once the installation steps are complete, clone the terratest-sample repository located at link to start executing teratest.
We’ll start by writing the test using Go and execute it.

First things first:

1.Your test file name should have _test in its name for example sample_test.go. This is how a Go looks for the test files.
2.Your test function name should start with Test with T being in capital letter. For example, TestFunction would work but testFunction will give you an error “no tests to run”.

Setup AWS auth configuration

We need AWS credentials to set up the infrastructure in AWS, we can configure it using environment variables or shared credentials file. Refer to the Terraform documentation for more details.

Terraform code for infrastructure can be found at the respective folder of the component For ec2, it’s under ec2_instance, and for API gateway, it’s under api_gateway folder. Terratest takes the output from Terraform’s output.tf as input for its tests.
Below is the snippet for testing if we have the same ssh key on ec2 instance we have used.

package terratest

import (
   "testing"
   "github.com/stretchr/testify/assert"
   "github.com/gruntwork-io/terratest/modules/terraform"
)

func TestEc2SshKey(t *testing.T) {
    terraformOptions := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
        TerraformDir: "../terraform",
    })
    defer terraform.Destroy(t, terraformOptions)
    terraform.InitAndApply(t, terraformOptions)
    ec2SshKey  := terraform.Output(t, terraformOptions, "instance_ssh_key")
    assert.Equal(t, "terratest", ec2SshKey)
}

We will divide it into different parts for proper understanding: In the first step we are defining a Go package named as terratest, and then we are importing different packages required for test execution.

package terratest

import (
   "testing"
   "github.com/stretchr/testify/assert"
   "github.com/gruntwork-io/terratest/modules/terraform"
)

Once we have all the prerequisites, we will create a function to execute actual test:

func TestEc2SshKey(t *testing.T) {
    terraformOptions := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
        TerraformDir: "../terraform",
    })
    defer terraform.Destroy(t, terraformOptions)
    terraform.InitAndApply(t, terraformOptions)
    ec2SshKey  := terraform.Output(t, terraformOptions, "instance_ssh_key")
    assert.Equal(t, "terratest", ec2SshKey)
}

With below section, we are defining directory where terratest should look for Terraform manifests i.e main.tf,output.tf for infrastructure creation.

 terraformOptions := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
     TerraformDir: "../terraform",
 })

In Go we use defer method to perform a cleanup task, it should be terraform destroy.
We are defining that using the below snippet:

defer terraform.Destroy(t, terraformOptions)
Now we can move forward to actual execution:

With terraform.InitAndApply we are invoking Terraform functions terraform init and apply which we generally use for Terraform execution:

   terraform.InitAndApply(t, terraformOptions)

As mentioned earlier, Terratest looks for output from output.tf for variable definition.

In the below snippet we are taking the ssh key from Terraform output and matching with ssh key name we have defined:

    ec2SshKey  := terraform.Output(t, terraformOptions, "instance_ssh_key")
    assert.Equal(t, "terratest", ec2SshKey)

Executing tests
Switch your directory to the location where you have cloned the repository. Navigate to the location where you have test files located.

Initialize Go modules, and download the dependencies. Take a look at Setting up your project section of Terratest documentation for more details.

go mod init ec2_instance
go mod tidy

And finally execute test:

$ go test -v

--- PASS: TestEc2SshKey (98.72s)
PASS
ok      command-line-arguments  98.735s

Let’s go a bit advance with Terratest

In the previous section, we have performed some basic level of testing using Terratest. Now, we will perform advanced test by deploying a API Gateway with Lambda and ALB as backend.

High-level functionality

GET request for API Gateway will be served by ALB and ANY method will be served by Lambda through API Gateway. After deployment, we will do a HTTP GET request against the gateway deployment URL and check if it’s returning success code.

Note: In our execution we are not using any API_KEY for authentication, but you should utilize it to replicate more realistic use of API Gateway.

Terraform output.tf

output "lb_address" {
  value = aws_lb.load-balancer.dns_name
  description = "DNS of load balancer"
}


output "api_id" {
  description = "REST API id"
  value       = aws_api_gateway_rest_api.api.id
}

output "deployment_invoke_url" {
  description = "Deployment invoke url"
  value       = "${aws_api_gateway_stage.test.invoke_url}/resource"
}

Code snippet for test execution

In the first scenario we have explained the basic syntax, so will directly go for test function.

func TestApiGateway(t *testing.T) {
    //awsRegion := "eu-west-2"
    terraformOptions := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
        TerraformDir: "../",
    })
    defer terraform.Destroy(t, terraformOptions)
    terraform.InitAndApply(t, terraformOptions)
    stageUrl := terraform.Output(t, terraformOptions,"deployment_invoke_url")
    time.Sleep(30 * time.Second)
    statusCode := DoGetRequest(t, stageUrl)
    assert.Equal(t, 200 , statusCode)
}

func DoGetRequest(t terra_test.TestingT, api string) int{
   resp, err := http.Get(api)
   if err != nil {
      log.Fatalln(err)
   }
   //We Read the response status on the line below.
   return resp.StatusCode
}

In the above snippet, we have defined a function DoGetRequest to run a HTTP GET test. Then we used the output of this function as an input for the TestApiGateway function.

Test execution and output

TestApiGateway 2022-03-01T06:56:18Z logger.go:66: deployment_invoke_url = "https://iuabeqgmj2.execute-api.eu-west-1.amazonaws.com/test/resource"
TestApiGateway 2022-03-01T06:56:18Z logger.go:66: lb_address = "my-demo-load-balancer-376285754.eu-west-1.elb.amazonaws.com"
TestApiGateway 2022-03-01T06:56:18Z retry.go:91: terraform [output -no-color -json deployment_invoke_url]
TestApiGateway 2022-03-01T06:56:18Z logger.go:66: Running command terraform with args [output -no-color -json deployment_invoke_url]
TestApiGateway 2022-03-01T06:56:19Z logger.go:66: "https://iuabeqgmj2.execute-api.eu-west-1.amazonaws.com/test/resource"
--- PASS: TestApiGateway (42.34s)
PASS
ok      command-line-arguments  42.347s

As you can see it has executed our test function TestApiGateway, in which it has performed HTTP GET test on deployment_invoke_url of API Gateway and returned test status.

Terratest modules extensibility and compliance testing using Terratest

We can utilize Terratest for compliance testing also. Some of the examples can be:

Check if Encryption is enabled on your SQS Queue or S3 bucket.
Verify if you have a particular throttling limit set for API gateway.
We have developed a Terratest check for API gateway.
In this example, we are verifying if we have Authorizer added for your API gateway. You can find out more on what are authorizers
at Document .

Currently Terratest does not have an API Gateway module in their AWS modules. You can find out available AWS modules at Terratest aws modules directory. Other Terratest modules such as Docker, Packer, Helm can be found at Terratest modules directory.

We have created our own test function for Authorizer using Terratest and AWS Go SDK methods at Function_URL.
More information on how to use AWS Go SDK: aws_go_sdk

Conclusion

Enterprises and their customers want products to be shipped faster. Infrastructure as Code provides just that with faster provisioning of infrastructure. As more and more infrastructure becomes code, the need for testing increases. In this post, we discussed how a tool like Terratest can help validate your code before you deploy it to production. We showed how Terratest works and even executed test cases to show how it’s done. One of the good things about Terratest is its extensibility that can be achieved by means of using modules as talked about in the post.

That’s all for this post. If you are working with Terratest or plan to use it and need some assistance, feel free to reach out to me via LinkedIn. We’re always excited to hear your thoughts and support!