Forem: Rahul Rai

OpenTelemetry in Action: Optimizing Database Operations

Rahul Rai — Fri, 13 May 2022 09:45:07 +0000

Many software developers can attest that some of the most significant issues in their applications arise from database performance. Though many developers prefer to use a relational database for enterprise applications, typical logging and monitoring solutions provide limited signals to detect database performance issues. Rooting out common bad practices such as chatty interactions between the application code and the database is non-trivial.

As developers, we need to understand how our database is performing from the context of user transactions. Ideally, we would have a common tool that can monitor the performance of both the application and the database concerning user transactions. OpenTelemetry has emerged as a popular tool for application monitoring, but it can also be extended for monitoring databases.

In this article, we’ll look at some common database performance issues to see how we can use OpenTelemetry to identify and fix them easily. For a hands-on learning experience, we’ll build a simple application that uses a SQL Server database. We’ll instrument the application with standard OpenTelemetry libraries and connect the application to an ingestion platform, Lightstep. Finally, we’ll use the ingested telemetry to surface our database issues, discussing the steps to resolve them.

The Basics of Observability and OpenTelemetry

If you’re unfamiliar with observability or OpenTelemetry, I recommend familiarizing yourself with the previous post in this “OpenTelemetry in Action” series. In that post, we cover the three types of information collected in an observable application (logs, metrics, and traces). We also look at the key components of the OpenTelemetry data model.

You can refer to this documentation for more in-depth coverage of how OpenTelemetry works.

Using OpenTelemetry to monitor databases

We noted how OpenTelemetry is used for instrumenting applications for observability and how it can be extended for us in monitoring databases. This monitoring is done through database clients rather than directly on the database server.

Due to access limitations or the nature of your platform, you may be restricted from installing monitoring libraries on a database server. Instead, you can use the OpenTelemetry instrumentation to monitor the database from the client-side. Though the instrumentation will not give you insights into the internals of the database, it will provide you with sufficient information to troubleshoot performance issues to improve the application user experience.

Using OpenTelemetry to Detect Database Performance Issues

Our project setup for detecting database performance issues is very similar to our setup in Part One, in which we used OpenTelemetry to identify database dependencies.

Again, we’ll use the .NET SQLClient instrumentation for OpenTelemetry and Lightstep for telemetry storage and analysis.

Exporting OTEL traces from .NET application

We’ll instrument our application with the OpenTelemetry SDK to emit observability signals. We’ll use the OpenTelemetry Protocol (OTLP) Exporter to send data to Lightstep, aggregating our traces and providing us with dashboards to analyze for insights.

Demonstration

For our demonstration, we’ll create a simple Employee Management Service (EMS) modeled as an ASP.NET Core minimal API. Our API has the following endpoints:

POST /ems/pay/{employeeId}: Calculates the pay of an employee based on the hours they logged on various projects. This endpoint will exhibit a chatty interaction with the database.
POST /ems/billing/pay-raise: Updates the pay of every employee earning under USD $300 to USD $300. This endpoint will exhibit querying a non-indexed field in the database.
POST /ems/payroll/remove/{employeeId}: Removes an employee from payroll. This endpoint will show how database locks affect the performance of queries.
POST /ems/add-employee/{employeeId}: Adds an employee to the payroll and timekeeping systems. This simulates how the performance of a business transaction spanning multiple services—and so, multiple database calls—can affect system performance.

The application is straightforward, but we’ve kept it concise to focus on instrumentation and the use of OpenTelemetry. For that reason, you won’t see coding best practices such as exception handling.

The application database consists of two tables: Payroll and Timekeeping, which save the employee pay rate and hours worked on a project.

The complete source code of this EMS application is available in this GitHub repository.

Spinning up the database

We start by spinning up a SQL server instance in docker:

docker run \
-e "ACCEPT_EULA=Y" \
-e "SA_PASSWORD=Str0ngPa$$w0rd" \
-p 1433:1433 \
--name monolith-db \
--hostname sql1 \
-d mcr.microsoft.com/mssql/server:2019-latest

Then, we create the EMS database and the tables used by our application, along with storing some seed data:

IF NOT EXISTS(SELECT * FROM sys.databases WHERE name = 'EMSDb')
BEGIN
    CREATE DATABASE EMSDb
END
GO

USE EMSDb

IF OBJECT_ID('[dbo].[Timekeeping]', 'U') IS NULL
BEGIN
    CREATE TABLE [Timekeeping] (
        [EmployeeId]      INT  NOT NULL,
        [ProjectId]       INT  NOT NULL,
        [WeekClosingDate] DATETIME NOT NULL,
        [HoursWorked]     INT  NOT NULL,
        CONSTRAINT [PK_Timekeeping] PRIMARY KEY CLUSTERED ([EmployeeId] ASC, [ProjectId] ASC,  [WeekClosingDate] ASC)
    )
END
GO

IF OBJECT_ID('[dbo].[Payroll]', 'U') IS NULL
BEGIN
    CREATE TABLE [Payroll] (
        [EmployeeId]   INT   NOT NULL,
        [PayRateInUSD] MONEY DEFAULT 0 NOT NULL,
        CONSTRAINT [PK_Payroll] PRIMARY KEY CLUSTERED ([EmployeeId] ASC)
    )
END
GO

TRUNCATE TABLE Payroll
TRUNCATE TABLE Timekeeping


INSERT INTO Payroll Values(1, 100)
INSERT INTO Payroll Values(2, 200)
INSERT INTO Payroll Values(3, 300)

INSERT INTO Timekeeping Values(1, 1111, GETDATE(), 10)
INSERT INTO Timekeeping Values(1, 2222, GETDATE(), 15)
INSERT INTO Timekeeping Values(2, 1111, GETDATE(), 15)
INSERT INTO Timekeeping Values(3, 2222, GETDATE(), 20)
GO

Implementing the API endpoints

Then, we write the code for the API endpoints. We replace the boilerplate code in the Program class with the following:

using System.Data.SqlClient;
using System.Diagnostics;
using Dapper;
using OpenTelemetry.Exporter;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

var lsToken = builder.Configuration.GetValue<string>("LsToken");

builder.Services.AddScoped(_ =>
    new SqlConnection(
      builder.Configuration.GetConnectionString("EmployeeDbConnectionString")
    )
);
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

var app = builder.Build();

app.UseSwagger();
app.UseSwaggerUI();

app.MapGet("/ems/pay/{empId}", async (int empId, SqlConnection db) =>
    {
        // op 1
        var payroll =
            await db.QuerySingleOrDefaultAsync<Payroll>("SELECT EmployeeId,PayRateInUSD FROM Payroll WHERE EmployeeId=@EmpId",
                new { EmpId = empId });

        // op 2
        var projects = await db.QueryAsync<Timekeeping>("SELECT EmployeeId,ProjectId,WeekClosingDate,HoursWorked FROM Timekeeping WHERE EmployeeId=@EmpId",
            new { EmpId = empId });

        var moneyEarned = projects.Sum(p => p.HoursWorked) * payroll.PayRateInUSD;
        return Results.Ok(moneyEarned);
    })
    .WithName("GetPayment")
    .Produces(StatusCodes.Status200OK);

app.MapPost("/ems/billing/pay-raise/", async (SqlConnection db) =>
    {
        var recordsAffected = await db.ExecuteAsync("UPDATE Payroll SET PayRateInUSD = 300 WHERE PayRateInUSD < 300");
        return Results.Ok(recordsAffected);
    })
    .WithName("Pay-Raise")
    .Produces(StatusCodes.Status200OK);

app.MapPost("/ems/payroll/remove/{empId}", async (int empId, SqlConnection db) =>
    {
        Payroll payrollRecord = new();
        async Task DeleteRecord()
        {
            db.Open();
            await using var tr = await db.BeginTransactionAsync();
            await db.ExecuteAsync("DELETE FROM Payroll WHERE EmployeeId=@EmpId", new { EmpId = empId }, tr);
            Thread.Sleep(5000);
            await tr.CommitAsync();
        }

        async Task GetRecord()
        {
            await using var db1 =
                new SqlConnection(builder.Configuration.GetConnectionString("EmployeeDbConnectionString"));
            Thread.Sleep(100);
            db1.Open();
            payrollRecord =
                await db1.QuerySingleOrDefaultAsync<Payroll>(
                    "SELECT EmployeeId,PayRateInUSD FROM Payroll WHERE EmployeeId=@EmpId", new { EmpId = empId });
            await db1.CloseAsync();
        }

        await Task.WhenAll(DeleteRecord(), GetRecord());

        return Results.Ok(payrollRecord);
    })
    .WithName("RemoveEmployeeFromPayroll")
    .Produces(StatusCodes.Status200OK);

app.MapPost("/ems/add-employee/{empId}", async (int empId, SqlConnection db) =>
    {
        //op 1
        await db.ExecuteAsync("INSERT INTO Payroll Values(@EmployeeId, @PayRateInUSD)",
            new Payroll { EmployeeId = empId, PayRateInUSD = 100 });

        // Simulate service call
        // Mock network call delay
        Thread.Sleep(1000);

        //op 2
        await db.ExecuteAsync(
            "INSERT INTO Timekeeping Values(@EmployeeId, @ProjectId, @WeekClosingDate, @HoursWorked)",
            new Timekeeping
            { EmployeeId = empId, HoursWorked = 0, ProjectId = 1, WeekClosingDate = DateTime.Today });

        return Results.Ok();    })
    .WithName("AddEmployee")
    .Produces(StatusCodes.Status201Created);

app.Run();


public class Timekeeping
{
    public int EmployeeId { get; set; }
    public int ProjectId { get; set; }
    public DateTime WeekClosingDate { get; set; }
    public int HoursWorked { get; set; }
}

public class Payroll
{
    public int EmployeeId { get; set; }
    public decimal PayRateInUSD { get; set; }
}

Now that our application code is in place, let’s automate the process of detecting the performance issues we have caused.

Adding instrumentation

We instrument the application with the OpenTelemetry SDK and SqlClient instrumentation library for .NET. First, we add the following NuGet package references to the API’s project file:

<PackageReference Include="OpenTelemetry" Version="1.2.0-rc2" />
<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.2.0-rc2" />
<PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.0.0-rc9" />
<PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc9" />
<PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.0.0-rc9" />
<PackageReference Include="OpenTelemetry.Instrumentation.SqlClient" Version="1.0.0-rc9" />

The SDK provides us with several extension methods that we can use to quickly plug OpenTelemetry into the request processing pipeline.

Next, we instrument OpenTelemetry in our application. This will also instrument the SqlClient to emit verbose telemetry. That telemetry will be key to surfacing database performance issues.

// Configure tracing
builder.Services.AddOpenTelemetryTracing(builder => builder
    // Customize the traces gathered by the HTTP request handler
    .AddAspNetCoreInstrumentation(options =>
    {
        // Only capture the spans generated from the ems/* endpoints
        options.Filter = context => context.Request.Path.Value?.Contains("ems") ?? false;
        options.RecordException = true;
        // Add metadata for the request such as the HTTP method and response length
        options.Enrich = (activity, eventName, rawObject) =>
        {
            switch (eventName)
            {
                case "OnStartActivity":
                {
                    if (rawObject is not HttpRequest httpRequest)
                    {
                        return;
                    }

                    activity.SetTag("requestProtocol", httpRequest.Protocol);
                    activity.SetTag("requestMethod", httpRequest.Method);
                    break;
                }
                case "OnStopActivity":
                {
                    if (rawObject is HttpResponse httpResponse)
                    {
                        activity.SetTag("responseLength", httpResponse.ContentLength);
                    }

                    break;
                }
            }
        };
    })
    // Customize the telemetry generated by the SqlClient
    .AddSqlClientInstrumentation(options =>
    {
        options.EnableConnectionLevelAttributes = true;
        options.SetDbStatementForStoredProcedure = true;
        options.SetDbStatementForText = true;
        options.RecordException = true;
        options.Enrich = (activity, x, y) => activity.SetTag("db.type", "sql");
    })
    .AddSource("my-corp.ems.ems-api")
    // Create resources (key-value pairs) that describe your service such as service name and version
    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("ems-api")
        .AddAttributes(new[] { new KeyValuePair<string, object>("service.version", "1.0.0.0") }))
    // Ensures that all activities are recorded and sent to exporter
    .SetSampler(new AlwaysOnSampler())
    // Exports spans to Lightstep
    .AddOtlpExporter(otlpOptions =>
    {
        otlpOptions.Endpoint = new Uri("https://ingest.lightstep.com:443/traces/otlp/v0.9");
        otlpOptions.Headers = $"lightstep-access-token={lsToken}";
        otlpOptions.Protocol = OtlpExportProtocol.HttpProtobuf;
    }));

Although the instrumentation is sufficient for us in its current state, let’s enrich the data further by adding relevant traces.

First, we define a tracer from which our application spans will originate.

var activitySource = new ActivitySource("my-corp.ems.ems-api");

Next, we create a span and add relevant details—attributes and events:

app.MapGet("/ems/pay/{empId}", async (int empId, SqlConnection db) =>
    {
        using var activity = activitySource.StartActivity("Chatty db operation", ActivityKind.Server);
        activity?.SetTag(nameof(Timekeeping.EmployeeId), empId);

        // op 1
        var payroll =
            await db.QuerySingleOrDefaultAsync<Payroll>("SELECT EmployeeId,PayRateInUSD FROM Payroll WHERE EmployeeId=@EmpId",
                new { EmpId = empId });

        // op 2
        var projects = await db.QueryAsync<Timekeeping>("SELECT EmployeeId,ProjectId,WeekClosingDate,HoursWorked FROM Timekeeping WHERE EmployeeId=@EmpId",
            new { EmpId = empId });

        var moneyEarned = projects.Sum(p => p.HoursWorked) * payroll.PayRateInUSD;
        return Results.Ok(moneyEarned);
    })
    .WithName("GetPayment")
    .Produces(StatusCodes.Status200OK);

We follow the same procedure to instrument the remaining endpoints.

Sending Instrumentation Data to Lightstep

To connect our application to Lightstep for data ingestion, we’ll need an API key. First, we create an account with Lightstep. Then, from the Project settings page, we copy the Token, which is our API key.

Your API key in Lightstep

We paste that token into our appsettings file.

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*",
  "ConnectionStrings": {
    "EmployeeDbConnectionString": "Server=localhost;Database=EMSDb;User Id=sa;Password=Str0ngPa$$w0rd;"
  },
  "LsToken": "<Lightstep token>"
}

Finding Common Database Issues

Our application is now ready. Let’s review our common database issues one by one.

Chatty/Sequential interaction with database

Let’s bring the /ems/pay/{empId} endpoint back into focus. From an examination of the code above, you’ll see that this endpoint makes two calls to the database, one after the other.

Non-optimal, chatty database calls slow down user transactions. Granted, you will encounter cases in which you need to read a record, make a decision based on the state of the record, and then update the record. In such cases, having multiple database calls is unavoidable. For fetching records, however, you can almost always use a single query. We launch the application (dotnet run) and send some requests to the /ems/pay/{empId} endpoint. Here is an example of a request that I sent to the endpoint:

Request to the pay endpoint

Next, we navigate to the Lightstep observability portal and click on the Operations tab to see all the spans Lightstep received from the application.

List of operations

We click on the /ems/pay/{employeeId} operation to view its end-to-end trace. By viewing the spans, we can ascertain the sequence of operations for any request, including database interactions. Clicking on the spans brings up its events and attributes, giving us deeper insight into the operation.

The final two spans visible in the trace are EMSDb, generated by our instrumented SQL Client. We click on the spans to view their attributes and events. They look like this:

Span captured from the first database operation

Span captured from the second database operation

From these details, we can gain some important insights:

The name of the database

The SQL statement used in the database operation
The type of SQL statement (text or stored procedure)
The hostname of the service which made the request
The duration of the database operation

The chatty behavior of the endpoint is very evident from the trace. A dead giveaway is the presence of two database operations in a read operation and the absence of custom traces between them to record any business logic. To resolve this issue, we need to combine the queries so that they produce a single resultset.

Unoptimized queries

Query performance is an important metric to monitor. If queries request a large amount of data (for example, by using the SELECT * approach or filtering data on non-indexed fields), then the performance of the user transactions will suffer. Let’s revisit the /emp/billing/pay-raise endpoint:

app.MapPost("/ems/billing/pay-raise/", async (SqlConnection db) =>
    {
        using var activity = activitySource.StartActivity("Non optimized query", ActivityKind.Server);
        var recordsAffected = await db.ExecuteAsync("UPDATE Payroll SET PayRateInUSD = 300 WHERE PayRateInUSD < 300");
        return Results.Ok(recordsAffected);
    })
    .WithName("Pay-Raise")
    .Produces(StatusCodes.Status200OK);

The query filters records on the PayRateInUSDfield, which we haven’t indexed. Although this issue will not be evident in a database with a small number of records, such queries in large databases will take a lot of time. The query time is apparent in the traces collected for the endpoint, which is shown below:

Span presenting the query duration

If the query duration exceeds the acceptable limit, you can inspect the operation further, indexing any fields used for filtering. This will speed up your database performance.

Database Locks

Database locks are one of the most complex issues to find because they only affect the operation waiting for a lock to be released. Meanwhile, the operation acquiring the lock proceeds unaffected. However, OpenTelemetry makes it easy to detect database locks because we can view the culprit operation and the operation waiting on the lock in the same trace. Let’s discuss the code behind our offending endpoint at /ems/payroll/remove/{empId}:

app.MapPost("/ems/payroll/remove/{empId}", async (int empId, SqlConnection db) =>
    {
        using var activity = activitySource.StartActivity("Db lock", ActivityKind.Server);
        activity?.SetTag(nameof(Timekeeping.EmployeeId), empId);

        Payroll payrollRecord = new();
        async Task DeleteRecord()
        {
            db.Open();
            await using var tr = await db.BeginTransactionAsync();
            await db.ExecuteAsync("DELETE FROM Payroll WHERE EmployeeId=@EmpId", new { EmpId = empId }, tr);
            Thread.Sleep(5000);
            await tr.CommitAsync();
        }

        async Task GetRecord()
        {
            await using var db1 =
                new SqlConnection(builder.Configuration.GetConnectionString("EmployeeDbConnectionString"));
            Thread.Sleep(100);
            db1.Open();
            payrollRecord =
                await db1.QuerySingleOrDefaultAsync<Payroll>(
                    "SELECT EmployeeId,PayRateInUSD FROM Payroll WHERE EmployeeId=@EmpId", new { EmpId = empId });
            await db1.CloseAsync();
        }

        await Task.WhenAll(DeleteRecord(), GetRecord());

        return Results.Ok(payrollRecord);
    })
    .WithName("RemoveEmployeeFromPayroll")
    .Produces(StatusCodes.Status200OK);

The DELETE operation begins a transaction but does not commit it for some time. This operation locks the record that will be deleted once the transaction is committed. After giving the DELETE transaction a little head start, we execute a SELECT operation to read the same record. The SELECT operation can’t proceed unless the DELETE operation releases the lock. This is clearly evident from the OpenTelemetry trace of this operation:

Operations causing database lock

If you were to investigate these operations individually, you might assume that the SELECT operation is the source of the performance issue. However, the aggregated spans of the database operations point to the sequential dependency between the DELETE and SELECT operations, which will prompt you to consider their relationship with one another.

The solution for the issue is to immediately commit a transaction without waiting for any lengthy operations to complete.

Business transactions spanning multiple services

If your user’s transactions span multiple services, then you should measure the response times of the different parts of the application and network. The /ems/add-employee/{empId} endpoint simulates a business transaction spread across two services as follows:

app.MapPost("/ems/add-employee/{empId}", async (int empId, SqlConnection db) =>
    {
        using var activity =
            activitySource.StartActivity("Multiple ops in a business transaction", ActivityKind.Server);
        activity?.SetTag(nameof(Timekeeping.EmployeeId), empId);

        //op 1
        await db.ExecuteAsync("INSERT INTO Payroll Values(@EmployeeId, @PayRateInUSD)",
            new Payroll { EmployeeId = empId, PayRateInUSD = 100 });

        // Simulate service call by creating another span
        using var innerActivity = activitySource.StartActivity("Second operation of business transaction", ActivityKind.Server);
        {
            // Mock network call delay
            Thread.Sleep(1000);

            //op 2
            await db.ExecuteAsync(
                "INSERT INTO Timekeeping Values(@EmployeeId, @ProjectId, @WeekClosingDate, @HoursWorked)",
                new Timekeeping
                    { EmployeeId = empId, HoursWorked = 0, ProjectId = 1, WeekClosingDate = DateTime.Today });
        }

        return Results.Ok();
    })
    .WithName("AddEmployee")
    .Produces(StatusCodes.Status201Created);

Let’s look at the trace generated for this operation, which clearly shows database operations for each service.

Business operations that requires communication between two services

The solution to such issues can vary in complexity. For a simple fix, you can optimize the network performance by placing the services physically close to one another (for example, in the same data center). A more complex approach would be to remodel the application to be eventually consistent so that you can asynchronously complete a transaction.

Database Exceptions

Our SQL client is instrumented to capture exceptions. Lightstep understands when spans carry exception details, highlighting such operations on the dashboard. We use the /ems/add-employee/{empId}endpoint to insert a duplicate record in the database, which throws an exception.

Lightstep highlights the exception in the Explorer window as follows:

Exceptions highlighted in explorer window

Clicking on the operation shows us the exception details captured as an event.

Exceptions details captured as event

We can use OpenTelemetry to record critical state changes and exceptions in the form of events, using the information to debug issues when they occur.

Conclusion

This article discussed how OpenTelemetry can be used to easily detect performance issues in an application database. The performance of an application database should be monitored in concert with user transactions. This helps us to avoid making unnecessary, low-value database optimizations.

Without altering the database in any manner, we instrumented a simple application to surface several types of database performance issues. With sufficient context, we can either quickly fix the underlying issue, or we can confidently develop a plan to improve the performance of our database.

OpenTelemetry in Action: Identifying Database Dependencies

Rahul Rai — Mon, 09 May 2022 01:14:35 +0000

Microservices can help any organization achieve its goal of increasing agility by addressing critical factors such as improving team autonomy, reducing time to market, cost-effectively scaling for load, and avoiding complete outages of the applications. As organizations break their monolith applications into microservices, one of the major hurdles they encounter is identifying database dependencies.

Database sharing can be a complex and time-consuming challenge to solve. Databases do not allow you to define what is shared and what is not. While modifying a schema to better serve one microservice, you might inadvertently break how another microservice uses that same database.

Additionally, it’s often difficult to identify the data owner and locate the business logic that manipulates the data.

In this article, we’ll explore how we might use OpenTelemetry to identify the components that share the same database and database objects, such as tables.

Observability and OpenTelemetry: The Basics

Before we build our demo application, let’s lay a foundation by discussing observability and OpenTelemetry.

What makes an application highly observable?

A system is said to be highly observable if its internal state can be inferred by studying its output at any point in time. For example, an observable mobile app that interacts with multiple services can reconstruct a transaction that produces an error response so that developers can identify the root cause of the failure.

Example of an observable application

An observable application collects three types of information for every transaction:

Logs: Record of individual events that make up the transaction.
Metrics: Record of aggregates of events that make up the transaction.
Traces: Record of the latency of operations to identify bottlenecks in the transaction.

To reconstruct the system’s state at any point in time, you need all three pieces of information.

What is OpenTelemetry?

OpenTelemetry is a system that generates logs, metrics, and traces in an integrated fashion. OpenTelemetry defines a standard to capture observability data. The OpenTelemetry data model has several key components.

Attributes

Every data structure in OpenTelemetry is comprised of attributes, which are key-value pairs. The OpenTelemetry standard defines what attributes any component (such as an SQL client or an HTTP request) can specify.

Events

Events are simply a timestamp and a set of attributes. You can record details such as messages and exception details against an event.

Context

Context comprises attributes that are common to a set of events. These are two types of contexts. Static context (or resource) defines the location of events. Their value doesn’t change after the application executable starts. Examples include the name or version of the service or the library name.

Dynamic context (or span) defines the active operation that contains the event. The value of span attributes changes when the operation executes. Some common examples of span attributes include the start time of a request, the HTTP response status code, or the path of the HTTP request.

In a distributed transaction, the context needs to be passed to all the associated services. In such cases the receiver service uses the context to produce new spans. A trace that crosses the service boundary becomes a distributed trace and the process of transferring context to other services is called context propagation.

Logs

Logs are events that only accompany resources. One example of this is the event emitted when a program starts up.

Traces

Events can be organized into a graph of operations associated with resources. A trace is the graph that presents the events related to a transaction.

Metrics

An event might occur several times in any application, or its value might change. A metric is an event whose value can be the count of related events or some computation of the value of the event. An example of a metric is the system memory event; its attributes are usage and utilization.

To learn about the concepts of OpenTelemetry in detail, refer to the documentation.

Using OpenTelemetry to Identify Database Dependencies

We discussed earlier that OpenTelemetry prescribes the attributes that the various components of an application should capture. The specification for attributes that should be part of the span covering a database client call is documented on GitHub. Many popular languages provide out-of-the-box instrumentation libraries that collect telemetry for database operations.

For this demo, we will use the .NET SQLClient instrumentation for OpenTelemetry in coordination with Lightstep for telemetry storage and analysis.

Let’s discuss the architecture of the demo application to understand the path that the telemetry takes to reach Lightstep. We’ll focus our discussion on traces alone, as they are sufficient to identify the dependency between databases and components of a monolith. However, any enterprise application should generate relevant logs and metrics in addition to traces for complete visibility.

Exporting OTEL traces from .NET application

First, we will instrument our monolith application with the OpenTelemetry SDK to emit observability signals. While instrumenting the application is a manual process for .NET applications, automatic instrumentation is available for applications built with languages such as Golang or Java.

We use the OpenTelemetry Protocol (OTLP) Exporter, which is included with the SDK. The exporter lets us send data directly to a telemetry ingestion service. OpenTelemetry platforms such as Jaeger and Lightstep aggregate the traces to help you draw insights.

Once integrated with the SDK, the various parts of your application, such as the ASP.NET Core request handler and SQL Client, automatically start producing traces with relevant information. Your code can generate additional traces to enrich the available information. In the case of .NET, the OpenTelemetry implementation is based on the existing types in the System.Diagnostics.* namespace as follows:

System.Diagnostics.ActivitySource represents an OpenTelemetry tracer responsible for producing Spans.
System.Diagnostics.Activity represents a Span.
You can add attributes to a span using the AddTag function. Also, you can add baggage using the AddBaggage function. The baggage is transported to the child activities, which might be available in other services using the W3C header.

After instrumenting your application, you can run automated tests or allow users to use your application to cover all the interaction paths between the application and database.

Demonstration

Let’s create a simple monolithic Employee Management Service (EMS) modeled as an ASP.NET Core minimal API. Our API will have the following endpoints:

POST /ems/billing: Records the hours worked by an employee on a project.
GET /ems/billing/{employeeId}: Fetches an employee’s hours on different projects.
POST /ems/payroll/add: Adds an employee to payroll.
GET /ems/payroll/{employeeId}: Fetches the payroll data for the employee.

You’ll notice that the monolith serves two distinct domains: billing and payroll. Such dependencies might not be very evident in complex monoliths, and segregating them might require significant code refactoring. However, by studying the dependencies, you can uncouple them with little effort.
The complete source code of the EMS application is available in this GitHub repository.

Spinning up the database

First, we start a SQL server instance in docker:

docker run \
-e "ACCEPT_EULA=Y" \
-e "SA_PASSWORD=Str0ngPa$$w0rd" \
-p 1433:1433 \
--name monolith-db \
--hostname sql1 \
-d mcr.microsoft.com/mssql/server:2019-latest

We use the following SQL script to create the EMS database and the tables used by our application:

IF NOT EXISTS(SELECT * FROM sys.databases WHERE name = 'EMSDb')
BEGIN
  CREATE DATABASE EMSDb
END
GO

USE EMSDb

IF OBJECT_ID('[dbo].[Timekeeping]', 'U') IS NULL
BEGIN
  CREATE TABLE [Timekeeping] (
    [EmployeeId]      INT  NOT NULL,
    [ProjectId]       INT  NOT NULL,
    [WeekClosingDate] DATETIME NOT NULL,
    [HoursWorked]     INT  NOT NULL,
    CONSTRAINT [PK_Timekeeping] PRIMARY KEY CLUSTERED ([EmployeeId] ASC, [ProjectId] ASC,  [WeekClosingDate] ASC)
  )
END
GO

IF OBJECT_ID('[dbo].[Payroll]', 'U') IS NULL
BEGIN
  CREATE TABLE [Payroll] (
    [EmployeeId]   INT   NOT NULL,
    [PayRateInUSD] MONEY DEFAULT 0 NOT NULL,
    CONSTRAINT [PK_Payroll] PRIMARY KEY CLUSTERED ([EmployeeId] ASC)
  )
END
GO

Implementing the API service

Next, we write the code for the API endpoints. We replace the boilerplate code in the Program class with the following code:

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddScoped(_ =>
    new SqlConnection(builder.Configuration.GetConnectionString("EmployeeDbConnectionString")));
var app = builder.Build();

app.UseSwagger();
app.UseSwaggerUI();

app.MapPost("/ems/billing", async (Timekeeping timekeepingRecord, SqlConnection db) =>
    {
        await db.ExecuteAsync(
            "INSERT INTO Timekeeping Values(@EmployeeId, @ProjectId, @WeekClosingDate, @HoursWorked)",
            timekeepingRecord);
        return Results.Created($"/ems/billing/{timekeepingRecord.EmployeeId}", timekeepingRecord);
    })
    .WithName("RecordProjectWork")
    .Produces(StatusCodes.Status201Created);

app.MapGet("/ems/billing/{empId}/", async (int empId, SqlConnection db) =>
    {
        var result = await db.QueryAsync<Timekeeping>("SELECT * FROM Timekeeping WHERE EmployeeId=@empId", empId);
        return result.Any() ? Results.Ok(result) : Results.NotFound();
    })
    .WithName("GetBillingDetails")
    .Produces<IEnumerable<Timekeeping>>()
    .Produces(StatusCodes.Status404NotFound);

app.MapPost("/ems/payroll/add/", async (Payroll payrollRecord, SqlConnection db) =>
    {
        await db.ExecuteAsync(
            "INSERT INTO Payroll Values(@EmployeeId, @PayRateInUSD)", payrollRecord);
        return Results.Created($"/ems/payroll/{payrollRecord.EmployeeId}", payrollRecord);
    })
    .WithName("AddEmployeeToPayroll")
    .Produces(StatusCodes.Status201Created);

app.MapGet("/ems/payroll/{empId}", async (int empId, SqlConnection db) =>
    {
        var result = await db.QueryAsync<Payroll>("SELECT * FROM Payroll WHERE EmployeeId=@empId", empId);
        return result.Any() ? Results.Ok(result) : Results.NotFound();
    })
    .WithName("GetEmployeePayroll")
    .Produces<IEnumerable<Payroll>>()
    .Produces(StatusCodes.Status404NotFound);

app.Run();


public class Timekeeping
{
    public int EmployeeId { get; set; }
    public int ProjectId { get; set; }
    public DateTime WeekClosingDate { get; set; }
    public int HoursWorked { get; set; }
}

public class Payroll
{
    public int EmployeeId { get; set; }
    public decimal PayRateInUSD { get; set; }
}

At this point, we can run the application, test the various endpoints, and look at the records saved in the database. Although the database dependencies of the various endpoints and request paths are clearly evident in this demonstration case, this would not be the case in large applications.

Next, let’s automate the process of discovering database dependencies.

Adding instrumentation

We instrument the application with the OpenTelemetry SDK and SqlClient instrumentation library for .NET. First, we add the following NuGet package references to the API’s project file:

<PackageReference Include="OpenTelemetry" Version="1.2.0-rc2" />
<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.2.0-rc2" />
<PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.0.0-rc9" />
<PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc9" />
<PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.0.0-rc9" />
<PackageReference Include="OpenTelemetry.Instrumentation.SqlClient" Version="1.0.0-rc9" />

The SDK provides us with several extension methods that we can use to quickly plug OpenTelemetry into the request processing pipeline.

The following piece of code instruments OpenTelemetry in our API. It will also instrument the SqlClient to emit verbose telemetry. The telemetry from the SqlClient is key to identifying the database dependencies in detail.

// Configure tracing
builder.Services.AddOpenTelemetryTracing(builder => builder
    // Customize the traces gathered by the HTTP request handler
    .AddAspNetCoreInstrumentation(options =>
    {
        // Only capture the spans generated from the ems/* endpoints
        options.Filter = context => context.Request.Path.Value?.Contains("ems") ?? false;
        options.RecordException = true;
        // Add metadata for the request such as the HTTP method and response length
        options.Enrich = (activity, eventName, rawObject) =>
        {
            switch (eventName)
            {
                case "OnStartActivity":
                {
                    if (rawObject is not HttpRequest httpRequest)
                    {
                        return;
                    }

                    activity.SetTag("requestProtocol", httpRequest.Protocol);
                    activity.SetTag("requestMethod", httpRequest.Method);
                    break;
                }
                case "OnStopActivity":
                {
                    if (rawObject is HttpResponse httpResponse)
                    {
                        activity.SetTag("responseLength", httpResponse.ContentLength);
                    }

                    break;
                }
            }
        };
    })
    // Customize the telemetry generated by the SqlClient
    .AddSqlClientInstrumentation(options =>
    {
        options.EnableConnectionLevelAttributes = true;
        options.SetDbStatementForStoredProcedure = true;
        options.SetDbStatementForText = true;
        options.RecordException = true;
        options.Enrich = (activity, x, y) => activity.SetTag("db.type", "sql");
    })
    .AddSource("my-corp.ems.ems-api")
    // Create resources (key-value pairs) that describe your service such as service name and version
    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("ems-api")
        .AddAttributes(new[] { new KeyValuePair<string, object>("service.version", "1.0.0.0") }))
    // Ensures that all activities are recorded and sent to exporter
    .SetSampler(new AlwaysOnSampler())
    // Exports spans to Lightstep
    .AddOtlpExporter(otlpOptions =>
    {
        otlpOptions.Endpoint = new Uri("https://ingest.lightstep.com:443/traces/otlp/v0.9");
        otlpOptions.Headers = $"lightstep-access-token={lsToken}";
        otlpOptions.Protocol = OtlpExportProtocol.HttpProtobuf;
    }));

Although the instrumentation is sufficient for us in its current state, let’s enrich the data further by adding relevant traces.

First, we define a tracer from which our application spans will originate.

var activitySource = new ActivitySource("my-corp.ems.ems-api");

Next, we create a span and add relevant details—attributes and events:

app.MapPost("/ems/billing", async (Timekeeping timekeepingRecord, SqlConnection db) =>
    {
        using var activity = activitySource.StartActivity("Record project work", ActivityKind.Server);
        activity?.AddEvent(new ActivityEvent("Project billed"));
        activity?.SetTag(nameof(Timekeeping.EmployeeId), timekeepingRecord.EmployeeId);
        activity?.SetTag(nameof(Timekeeping.ProjectId), timekeepingRecord.ProjectId);
        activity?.SetTag(nameof(Timekeeping.WeekClosingDate), timekeepingRecord.WeekClosingDate);

        await db.ExecuteAsync(
            "INSERT INTO Timekeeping Values(@EmployeeId, @ProjectId, @WeekClosingDate, @HoursWorked)",
            timekeepingRecord);
        return Results.Created($"/ems/billing/{timekeepingRecord.EmployeeId}", timekeepingRecord);
    })
    .WithName("RecordProjectWork")
    .Produces(StatusCodes.Status201Created);

We follow the same procedure to instrument the remaining endpoints.

Connecting to Lightstep

Finally, we need an API key to send traces to Lightstep. We start by creating an account. From the Project settings page of our account, we find the Token, which will serve as our API key.

Your API key in Lightstep

We copy the token and paste it in the appsettings file.

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*",
  "ConnectionStrings": {
    "EmployeeDbConnectionString": "Server=localhost;Database=EMSDb;User Id=sa;Password=Str0ngPa$$w0rd;"
  },
  "LsToken": "<Lightstep token>"
}

Sending requests

Our application is now ready. We launch the application and send some requests to each endpoint. Here is an example of a request that I sent to the /ems/billing endpoint. This request should create a record in the **Timekeeping **table of the database.

Sending request to the billing endpoint

Here’s another request I made to the /emp/payroll/add endpoint to add a record to the Payroll table:

Sending request to the payroll endpoint

When we navigate to the Lightstep observability portal, we can click on the Operations tab to see all the spans Lightstep received from the application.

View spans from the application in Lightstep

When we click on the /ems/payroll/add operation, we can view the end-to-end trace. By viewing the spans, we can ascertain the sequence of operations for any request. Clicking on the spans brings up its events and attributes, from which we can gain deeper insights into the operation. The final span visible in the trace is EMSDb, which was generated by our instrumented SQL Client. We click on the span to view its attributes and events as follows:

Details of the span generated by the billing endpoint

We can draw some key insights from the attributes:

The name of the database
The SQL statement used in the database operation
The type of SQL statement (text or stored procedure)
The hostname of the service that made the request

We find a similar set of details from the child span of the /ems/billing operation.

Details of the span generated by the billing endpoint

By combing the information from the traces, we can infer the following:

Ingress operations (operations that receive the external request)
The sequence of activities to fulfill a request, including external service calls and database operations.
Database operations involved in each operation.

Together, this information is sufficient for us to plan the segregation of services and databases and establish contacts for communication between the microservices.

Conclusion

This article discussed one of the common challenges that developers encounter while transitioning their monolith applications to microservices. Of all the concerns, splitting a database is a complex endeavor because any service that has access to the database can manipulate it.

By using OpenTelemetry, we can identify the dependencies among the various components and between the components and the database. With the knowledge and understanding of our dependencies, we can develop the refactoring plan for our components, planning how they should evolve as independent microservices.

Practical Introduction to Kubernetes Autoscaling Tools with Linode Kubernetes Engine

Rahul Rai — Sun, 06 Mar 2022 11:21:33 +0000

Your cloud infrastructure can scale in real time with your application without making a configuration change or writing a line of code. Autoscaling is the process of increasing or decreasing the capacity of application workloads without human intervention. When tuned correctly, autoscaling can reduce costs and engineering toil in maintaining the applications.

The overall process of enabling autoscaling is simple. It begins with determining the set of metrics that can provide an indicator for when Kubernetes should scale the application capacity. Next, a set of rules determines whether the application should be scaled up or down. Finally, using the Kubernetes APIs, the resources available to the application are expanded or contracted to accommodate the work that the application must perform.

Autoscaling is the process of increasing or decreasing the capacity of application workloads without human intervention. When tuned correctly, autoscaling can reduce costs and engineering toil in maintaining the applications. The overall process of autoscaling is simple. It begins with determining the set of metrics that can provide an indicator for when Kubernetes should scale the application capacity. Next, a set of rules determine whether the application should be scaled up or down. Finally, using the Kubernetes APIs, the resources available to the application are expanded or contracted to accommodate the work that the application must perform.

Autoscaling is a complex process, and it serves some categories of applications better than others. For example, if an application’s capacity requirements don’t change often, you will be better off provisioning resources for the highest traffic that the application will handle. Similarly, if you can predict the application load reliably, you can manually adjust the capacity at those times rather than investing in an autoscaling solution.

Apart from the variable load on the application, other primary motivations for autoscaling include managing costs and capacity. For example, cluster autoscaling allows you to save money on public clouds by adjusting the number of nodes in your cluster. Also, if you have a static infrastructure, autoscaling will enable you to dynamically manage the allocation of capacity to your workloads so that you can optimally utilize your infrastructure.

At a high level, autoscaling can be divided into two categories:

Workload autoscaling: Dynamically managing capacity allocation to individual workloads.
Cluster autoscaling: Dynamically managing the cluster’s capacity.

Let’s first dive into the details of scaling workloads in Kubernetes. Some of the standard tools used for autoscaling workloads on Kubernetes are Horizontal Pod Autoscaler (HPA), Vertical Pod Autoscaler (VPA), and Cluster Proportional Autoscaler (CPA). To work with the autoscalers, we need a cluster and a simple test application, which we will set up next.

Creating a Linode Kubernetes Engine Cluster

Linode offers a managed Kubernetes offering known as Linode Kubernetes Engine (LKE). Getting started is easy: sign up for a free Linode account and follow the LKE onboarding guide to create your cluster.

For this tutorial, I created a cluster consisting of two nodes (called Linodes), each with 2 CPU cores and 4 GB of memory as follows:

LKE Cluster

To work with the cluster, you need the cluster’s kubeconfig file, which you can download from the cluster overview section. There are several strategies you can use to merge the kubeconfig files. However, I prefer to update the KUBECONFIG environment variable with the path to the kubeconfig file.

Let’s now build a simple application that we will use to test the various autoscalers.

Pressure API

The Pressure API is a simple .NET REST API that allows you to apply CPU and memory pressure on the pod in which the application is running via its two endpoints:

/memory/{numMegaBytes}/duration/{durationSec}: This endpoint will add the specified number of Megabytes to memory and maintain the pressure for the specified duration.
/cpu/{threads}/duration/{durationSec}: This endpoint will run the specified number of threads on the CPU and maintain the pressure for the specified duration.

Following is the complete source code of the application:

using System.Xml;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

var app = builder.Build();

if (app.Environment.IsDevelopment())
{
    app.UseSwagger();
    app.UseSwaggerUI();
}

app.MapPost("/memory/{numMegaBytes}/duration/{durationSec}", (long numMegaBytes, int durationSec) =>
    {
        // ReSharper disable once CollectionNeverQueried.Local
        List<XmlNode> memList = new();

        try
        {
            while (GC.GetTotalMemory(false) <= numMegaBytes * 1000 * 1000)
            {
                XmlDocument doc = new();
                for (var i = 0; i < 1000000; i++)
                {
                    memList.Add(doc.CreateNode(XmlNodeType.Element, "node", string.Empty));
                }
            }
        }
        // Don't fail if memory is not available
        catch (OutOfMemoryException ex)
        {
            Console.WriteLine(ex);
        }

        Thread.Sleep(TimeSpan.FromSeconds(durationSec));
        memList.Clear();
        GC.Collect();
        GC.WaitForPendingFinalizers();
        return Results.Ok();
    })
    .WithName("LoadMemory");

app.MapPost("/cpu/{threads}/duration/{durationSec}", (int threads, int durationSec) =>
    {
        CancellationTokenSource cts = new();
        for (var counter = 0; counter < threads; counter++)
        {
            ThreadPool.QueueUserWorkItem(tokenIn =>
            {
#pragma warning disable CS8605 // Unboxing a possibly null value.
                var token = (CancellationToken)tokenIn;
#pragma warning restore CS8605 // Unboxing a possibly null value.
                while (!token.IsCancellationRequested)
                {
                }
            }, cts.Token);
        }

        Thread.Sleep(TimeSpan.FromSeconds(durationSec));
        cts.Cancel();
        Thread.Sleep(TimeSpan.FromSeconds(2));
        cts.Dispose();
        return Results.Ok();
    })
    .WithName("LoadCPU");


app.Run();

You don’t need to worry about the details of the application. I have published the container image of the application on GitHub Packages that you can use in your K8s specifications that we will build in the subsequent sections.

You can download the source code of the application and the other artifacts of this tutorial from the following GitHub repository:

The Kubernetes specs used in this tutorial are available in the spec folder of the code repository. Use the following spec to deploy the application to your LKE cluster:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pressure-api-deployment
spec:
  selector:
    matchLabels:
      app: pressure-api
  replicas: 1
  template:
    metadata:
      labels:
        app: pressure-api
    spec:
      containers:
        - name: pressure-api
          image: ghcr.io/rahulrai-in/dotnet-pressure-api:latest
          ports:
            - containerPort: 80
          resources:
            limits:
              cpu: 500m
              memory: 500Mi
---
apiVersion: v1
kind: Service
metadata:
  name: pressure-api-service
  labels:
    run: php-apache
spec:
  ports:
    - port: 80
  selector:
    app: pressure-api

Your application is now ready to accept requests but can be accessed only within the cluster. We will later use an ephemeral pod to send requests to our API. Let’s now discuss the most common autoscaler from the autoscaler family: Horizontal Pod Autoscaler (HPA).

Horizontal Pod Autoscaler (HPA)

Horizontal Pod Autoscaler allows you to dynamically adjust the number of pods in your cluster based on the current load. Kubernetes natively supports it with the HorizontalPodAutoscaler resource and controller bundled into the kube-controller-manager. The HPA relies on Kubernetes Metrics Server to provide the PodMetrics. The Metrics Server collects CPU and memory usage of the pods from the kubelets running on each node in the cluster and makes them available to the HPA through the Metrics API.

The following diagram illustrates the components involved in the process:

Horizontal Pod Autoscaling

The Metrics Server polls the Summary API endpoint of the kubelet to collect the resource usage metrics of the containers running in the pods. The HPA controller polls the Metrics API endpoint of the Kubernetes API server every 15 seconds (by default), which it proxies to the Metrics Server. In addition, the HPA controller continuously watches the HorizontalPodAutoscaler resource, which maintains the autoscaler configurations. Next, the HPA controller updates the number of pods in the deployment (or other configured resource) to match the requirements based on the configurations. Finally, the Deployment controller responds to the change by updating the ReplicaSet, which changes the number of pods.

We know that the Metrics server is a prerequisite for HPA and VPA. Follow the instructions mentioned in the official Metrics Server guide to install it on your cluster. If you face TLS issues with the installation, use the metrics-server.yaml spec available in the spec folder of the code repo as follows:

kubectl apply -f spec/metrics-server.yaml

Let’s now configure the HorizontalPodAutoscaler object to scale out our deployment to five replicas and scale it down to one replica based on the average utilization of the memory resource as follows:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: pressure-api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: pressure-api-deployment
  minReplicas: 1
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 40

If the average utilization of the memory stays at over 40%, HPA will increase the replica count and vice versa. You can extend the rule to include CPU utilization as well. In this case, the HPA controller will determine the maximum number of replicas based on the combination of the rules and use the highest value.

Before we begin, let’s watch the HPA and the deployment in two different terminal windows to see the changes in the replica count in real-time.

kubectl get hpa pressure-api-hpa --watch

kubectl get deployment pressure-api-deployment --watch

To trigger the HPA, we will spin up an ephemeral pod and instruct it to send requests to the /memory/{numBytes}/duration/{durationSec} endpoint. The following command will trigger the HPA to scale up the pods to reduce the memory pressure.

kubectl run -i --tty mem-load-gen --rm --image=busybox --restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q -O- --post-data= http://pressure-api-service/memory/1000/duration/180; done"

You can watch the HPA updating the replica count of the deployment in the terminal window. Note the growth of the active utilization against the target as follows:

HPA in action

Simultaneously, you can see the replicas getting updated as follows:

Increase in replica count triggered by HPA

There are a few considerations to keep in mind when using HPA:

Your application should be capable of sharing load among distinct instances.
Your cluster should have sufficient capacity to accommodate the expansion of the number of pods. This can be addressed by provisioning the required capacity ahead of time and using alerts to prompt your platform operators to add more capacity to the cluster. You can also use cluster autoscaling to scale the cluster automatically. We will discuss this feature later in this tutorial.
CPU and memory might not be the right metrics for your application to make scaling decisions. In such cases, you can use HPA (or VPA) with custom metrics as an alternative. To use custom metrics for autoscaling, you can use a custom metrics adapter instead of the Kubernetes Metrics Server. Popular custom metrics adapters are the Prometheus adapter and Kubernetes Event-Driven Autoscaler (KEDA).

Before we proceed, delete the HPA that you just created and reset the replica count of the deployment as follows:

kubectl delete hpa/pressure-api-hpa

kubectl scale --replicas=2 deployment/pressure-api-deployment

Let’s discuss another type of autoscaler available in Kubernetes: Vertical Pod Autoscaler (VPA).

Vertical Pod Autoscaler (VPA)

Vertical Pod Autoscaler allows you to adjust the resource capacity of a single instance dynamically. In the context of pods, this involves changing the amount of CPU and memory resources available to the pod. Unlike HPA, which is included in the core Kubernetes, the VPA requires you to install three controller components in addition to the Metrics Server. The following diagram illustrates Kubernetes components and their interactions with the VPA:

Vertical Pod Autoscaler

Recommender: Determines the optimum CPU and memory values based on the usage of the pod resources.
Admission plug-in: Mutates the pod’s resource requests and limits when the pod is created based on the Recommender’s recommendation.
Updater: Evicts pods so that the admission plug-in intercepts its recreation request.

Follow the installation instructions in the ReadMe guide of VPA to prepare your cluster. Once the installation is complete, you can verify the health of the VPA components by running the following command:

kubectl get pods -l "app in (vpa-recommender,vpa-admission-controller,vpa-updater)" -n kube-system

Health of VPA pods

Let’s understand how the VPA’s scaling operation works. The resource requests declarations in pod specification ensure that Kubernetes reserves the minimum required resources for the pod. When VPA detects that pods are close to their resource consumption limits, it will automatically compute a new, more appropriate set of values. If you define both the resource requests and the resource limits in the pod specification, VPA will maintain the request:limit ratio when updating the values. Thus whenever VPA updates the resource requests, it will also change the resource limits.

We’ll define a VPA policy to automatically adjust the CPU and memory requests without adding more pods to process workload as follows:

apiVersion: "autoscaling.k8s.io/v1"
kind: VerticalPodAutoscaler
metadata:
  name: pressure-api-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: pressure-api-deployment
  updatePolicy:
    updateMode: Recreate
  resourcePolicy:
    containerPolicies:
      - containerName: "*"
        minAllowed:
          cpu: 0m
          memory: 0Mi
        maxAllowed:
          cpu: 1
          memory: 2000Mi
        controlledResources: ["cpu", "memory"]
        controlledValues: RequestsAndLimits

The specification will apply to all containers of the deployment. The minimum and maximum threshold values will ensure that VPA operates within a reasonable range. The controlledResources field specifies the resources that will be autoscaled by the VPA.

The VPA supports four update modes. The Recreate and Auto modes are the only ones that activate autoscaling. However, there are limited use cases for these. The Initial mode will apply admission control to the set resource values when they are created, but it will prevent the Updater from evicting any pods. The most useful mode is the Off mode. In this mode, the VPA will not scale the resources. However, it will recommend the resource values. You can use this mode to compute the optimal resource values for your application during their comprehensive load testing and profiling before they go to production. The recommended values can be applied to the production deployment specification saving engineering toil.

Apply the previous specification and execute the following command to watch the autoscaler:

kubectl get vpa/pressure-api-vpa --watch

We will now apply CPU pressure using the following command to activate the VPA:

kubectl run -i --tty mem-load-gen --rm --image=busybox --restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q -O- --post-data= http://pressure-api-service/cpu/10/duration/180; done"

After some time, execute the following command to view the recommendations produced by the VPA.

kubectl describe vpa/pressure-api-vpa

Following is a snipped output of the previous command that presents the recommendations from the VPA:

Recommendations from VPA

Use the Target value as a baseline recommendation for the CPU and memory requests. If the upper and lower bounds defined in the VPA spec are not optimal, use the Uncapped Target as the baseline, representing the target estimation produced with no minAllowed and maxAllowed restrictions.

Since we enabled vertical autoscaling, the newly created pods will have VPA annotations applied by the admission controller. The following command will display the annotations of the pod:

kubectl get pod  <pod name> -o jsonpath='{.metadata.annotations}'

Here is the output of the previous command (from the K9s console):

Pod annotations

Let’s delete the autoscaler and reset our deployment before moving to the next scaler in the list.

kubectl delete vpa/pressure-api-vpa
kubectl scale --replicas=1 deployment/pressure-api-deployment

Cluster Proportional Autoscaler (CPA)

The Cluster Proportional Autoscaler (CPA) is a horizontal pod autoscaler that scales replicas based on the number of nodes in the cluster. Unlike other autoscalers, it does not rely on the Metrics API and does not require the Metrics Server. Additionally, unlike other autoscalers we saw, a CPA is not scaled with a Kubernetes resource but instead uses flags to identify target workloads and a ConfigMap for scaling configuration. The following diagram illustrates the components of the CPA:

Cluster Proportional Autoscaler

CPA has relatively limited use cases. For example, CPA is generally used to scale-out platform services such as cluster DNS, which needs to scale with the workload deployed on the cluster. Another use case of CPA is to have a simple mechanism to scale out workloads since it does not require using Metrics Server or Prometheus Adapter.

You can install CPA on your cluster using its Helm chart. Use the following command to add the cluster-proportional-autoscaler Helm repo as follows:

helm repo add cluster-proportional-autoscaler https://kubernetes-sigs.github.io/cluster-proportional-autoscaler
helm repo update

You can define the autoscaling rules in the chart’s values file, which creates a ConfigMap with the specified configurations. You can later edit the ConfigMap to alter the behavior of the autoscaler without having to reinstall the chart.

Create a file named cpa-values.yaml and add the following content:

config:
  ladder:
    nodesToReplicas:
      - [1, 3]
      - [2, 5]
options:
  namespace: default
  target: "deployment/pressure-api-deployment"

You can specify one of the two scaling methods used by the CPA:

Linear: Scales your application in direct proportion to how many nodes or cores are in the cluster.
Ladder: Uses a step function to determine the ratio of nodes:replicas and/or cores:replicas.

In the above example, CPA will scale the deployment to three replicas if we have one node in the cluster and five replicas for two nodes. Let’s now install the chart and supply the configuration to it.

helm upgrade --install cluster-proportional-autoscaler \
    cluster-proportional-autoscaler/cluster-proportional-autoscaler --values cpa-values.yaml

As soon as you install CPA, you will find that it scales the pressure-api-deployment deployment to 5 replicas since our cluster has two nodes.

Let’s delete the CPA before moving to the next autoscaler in the list as follows:

helm delete cluster-proportional-autoscaler

We’ve looked at several approaches of autoscaling workloads using the core Kubernetes and community-built add-on components. Next, we will discuss how you can scale the Kubernetes cluster itself.

Cluster Autoscaler (CA)

Manually adding and removing capacity from the Kubernetes cluster can significantly drive up the cluster management costs and engineering toil. Cluster Autoscaler automates adding and removing worker nodes to the cluster to meet the desired capacity. CA works nicely in conjunction with the HPA. As soon as HPA starts approaching the compute resources limit, CA can calculate the number of nodes to satisfy the shortage and add new nodes to the cluster. Also, when CA determines that nodes have been underutilized for an extended period, it can reschedule the pods to other nodes and remove the underutilized nodes from the cluster.

The implementation of Cluster Autoscaler varies with the cloud providers. Some cloud providers such as Azure and AWS support Cluster API. Cluster API uses its Kubernetes operator to manage cluster infrastructure. The Cluster Autoscaler offloads the operation to update the node count to the Cluster API controller. Cluster autoscaling can be helpful if you consider the following before implementing it:

Ensure that you understand how your application will behave under load and remove the bottlenecks that prevent the application from scaling horizontally.
Know the upper scaling limit that the cloud provider might enforce.
Understand the speed at which the cluster can scale when the need arises.

Enabling Cluster autoscaling on LKE is straightforward. First, navigate to the cluster overview page and click on the Autoscale Pool button.

LKE cluster overview page

Then, in the following dialog, enter the minimum and the maximum number of nodes that LKE should maintain as follows:

Enable LKE cluster autoscaling

The LKE cluster autoscaler responds to Pending pods that couldn’t be scheduled due to insufficient compute resources. The autoscaler monitors the underutilized nodes and removes them from the cluster to scale down the cluster.

We started with a two-node cluster, each with two CPU cores and 4 GB of memory. To trigger the cluster autoscaler, we can add more replicas to our application as follows:

kubectl scale --replicas=15 deployment/pressure-api-deployment

After executing the command, you will find that several pods of the deployment reach the pending state as follows:

Pods waiting to be scheduled

Soon afterward, LKE adds more nodes to the cluster and schedules some of them on the new nodes as follows:

LKE scaling out the cluster

You will find that some of the pods are still in the pending state because we instructed LKE to scale out to a maximum of four nodes. Finally, to clean up the environment, execute the following command:

kubectl delete deployment/pressure-api-deployment

Summary

We discussed the concept of horizontal autoscaling, vertical autoscaling, and cluster autoscaling, along with their use cases and considerations. If your applications are often subject to changes in capacity requirements, you can use HPA to scale them horizontally. VPA may help you to identify the optimal resource values for your applications. CPA can help you address the scaling requirements of applications that need to scale with the workload in the cluster. If your workloads can scale beyond the cluster’s capacity, use CA to autoscale the cluster itself. If you’re considering a managed Kubernetes service like LKE, look for a solution that has built-in autoscaling tools to reduce your effort.

Enhancing Istio Operations with Kong Istio Gateway

Rahul Rai — Sun, 13 Feb 2022 10:06:21 +0000

If you’re a developer for a service-oriented application, routing requests between services can be overwhelming. This work may force you to focus on operational details that take you away from building great features for your customers.

Fortunately, with Kong Istio Gateway, we can solve many inter-service networking concerns such as security, resiliency, observability, and traffic control with services-first networking policies. By offloading network-related problems to the service mesh, you can focus on building features that deliver business value.

In this article, we’ll look more closely at Kong Istio Gateway, exploring its benefits and walking through key use cases for enhancing Istio with Kong Gateway.

Kong Istio Gateway Overview

By default, an Istio mesh in a Kubernetes cluster allows for communication between services within the cluster. To expose the services in the mesh to external traffic, Kubernetes supports an ingress controller named Kubernetes Ingress.

Istio comes with its own implementation of an ingress controller known as a gateway. Kong Istio Gateway is a drop-in replacement of the Istio ingress gateway. We can easily extend Kong with a wide range of enterprise-grade plugins that address a variety of Layer 4 to Layer 7 application concerns such as authentication, traffic routing, and security at the gateway level.

Kong Istio Gateway is deployed as a regular service (of LoadBalancer type) in your service mesh, enabling you to use the entire suite of Istio APIs. The following diagram presents the logical path that an external request takes to reach the destination service in the presence of Kong Istio Gateway.

Figure 1: Logical flow of request in Istio

The request originating outside the cluster first meets Kong Istio Gateway. The Ingress controller supplies the gateway’s request-routing rules, which the gateway uses to route the request to the associated service. For communication within the mesh, the virtual service sends the request, which uses the destination’s rules in the form of layer 7 attributes to determine which service subset should receive the request.

Benefits of Using Kong Istio Gateway

You can easily extend the features of Kong Istio Gateway using plugins. The combination of plugins and ingress rules grants you absolute control over your application’s requests and response pipeline. The following diagram illustrates the components involved in routing a client request to the destination service.

Figure 2: Components involved in routing

There are a few key advantages of using Kong Istio Gateway over the Istio ingress gateway:

Ability to add indirection in enforcing enterprise policies : Kong’s plugin hub provides a variety of plugins that the operations team can independently install without affecting the development process. We can use the plugins to quickly enforce organization-wide policies, such as JWT authentication.
Support for application-level policies : Istio gateway lets you configure routing rules at level 4 to level 6 of the network stack. For example, you can specify request routing rules based on attributes such as the port, host, TLS key, and certificates. Kong Gateway enables you to enforce application-level policies such as validating requests.
Build custom request and response pipelines : Using Kong request-response transformation plugins, you can alter the request and response pipeline to support client requirements.

The ability to add plugins to Kong Gateway allows you to keep your application’s architecture flexible so that you can iterate quickly in the face of changing requirements.

Enhancing Istio Operations

The Istio documentation website outlines some key use cases of Istio:

Traffic Management : Includes capabilities such as request routing, fault injection, traffic shifting, circuit breaking, and mirroring.
Security : Includes certificate management, authorization, and authentication capabilities.
Policy Enforcement : Using Envoy to enforce rate limits.
Observability : Automatic collection of metrics, traces, and logs from services. It also includes support for out-of-box visualization tools such as Kiali.

Istio specializes in inter-service communication. Like other services in the service mesh, Kong Istio Gateway gets Envoy sidecar pods linked to its pods. This makes it strongly embedded in the service mesh. Kong has a rich set of capabilities to manage the north-south traffic (traffic between the client and the service). We will use these capabilities to enhance the inter-service communication (east-west traffic) capabilities of Istio.

Demo Application: HTTP Echo

We will use a very basic application that echoes any HTTP request. This is the HTTP version of the popular TCP echo sample application of Istio. I’ve used Docker Desktop with Kubernetes enabled for this demo. You’ll also need to have helm installed.

Here is the source code of the application:

package main

import (
    "net/http"
    "os"
    "time"
)

var prefix string

func main() {
    port := os.Args[1]
    prefix = os.Args[2]
    http.HandleFunc("/", echoHandler)
    http.ListenAndServe(":"+port, nil)
}

func echoHandler(writer http.ResponseWriter, request *http.Request) {
    request.Write(writer)
    writer.Write([]byte(prefix + " says OK at " + time.Now().String()))
}

Use the following Dockerfile to build the container image of the application:

FROM golang:1.15.7-buster
ADD main.go /go/src/main.go
ENTRYPOINT [ "go", "run", "/go/src/main.go" ]
CMD [ "9000", "hello" ]
EXPOSE 9000

Next, use the following command to build the container image from the specification.

docker build -t http-echo .

You can either use the container image you built or the container image I published on the DockerHub registry. You can also inspect the application source code in the GitHub repository.

Install Kong Istio Gateway

Follow the Istio installation guide instructions to download the Istio binaries and the Istio CLI tool: istioctl. We need just the Istio daemon, IstioD, for enabling Kong Istio Gateway. Use the following command to install just the IstioD component on your cluster:

istioctl install --set profile=minimal -y

Let’s now deploy Kong Istio Gateway in our cluster. We will label the namespace of the gateway so that the gateway pod gets a sidecar and becomes a part of the service mesh. We will use the official Kong Helm chart to install Kong Istio Gateway in the cluster as follows:

kubectl create namespace kong-istio
kubectl label namespace kong-istio istio-injection=enabled
helm repo add kong https://charts.konghq.com && helm repo update
helm install -n kong-istio kong-istio kong/kong

We previously discussed how Kong Gateway gets deployed as a LoadBalancer service in the cluster. The service provides an external endpoint where you can send HTTP requests. Execute the following command to get the endpoint of the service.

kubectl get svc/kong-istio-kong-proxy -n kong-istio

Install Service Mesh Visualization Tools

With Kong Gateway, we deployed the service mesh to manage our application. An advantage of having the gateway as part of the mesh is that we can use the excellent Istio mesh visualization tools to get a complete picture of the service topology. Let’s now install Prometheus and Kiali. Later, we will use them to inspect the topology of our mesh using the following commands:

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.12/samples/addons/prometheus.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.12/samples/addons/kiali.yaml

Install HTTP Echo Application

Now, let’s deploy our service to the cluster and add it to the mesh. We will deploy two versions of the application that only differ in the response they produce on request. Create a YAML specification file named echo-service.yaml for the HTTP Echo service. Remember to replace the name of the container image based on the container registry that you use (local or DockerHub):

kind: Namespace
apiVersion: v1
metadata:
  name: echo
  labels:
    istio-injection: enabled
---
apiVersion: v1
kind: Service
metadata:
  name: http-echo
  namespace: echo
  labels:
    app: http-echo
    service: http-echo
spec:
  ports:
    - port: 8080
      name: http
  selector:
    app: http-echo
    version: v1
---
apiVersion: v1
kind: Service
metadata:
  name: http-echo-2
  namespace: echo
  labels:
    app: http-echo
    service: http-echo-2
spec:
  ports:
    - port: 8080
      name: http
  selector:
    app: http-echo
    version: v2
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: http-echo-v1
  namespace: echo
  labels:
    app: http-echo
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: http-echo
      version: v1
  template:
    metadata:
      labels:
        app: http-echo
        version: v1
    spec:
      containers:
        - name: http-echo
          image: http-echo:latest
          imagePullPolicy: IfNotPresent
          args: ["8080", "one"]
          ports:
            - containerPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: http-echo-v2
  namespace: echo
  labels:
    app: http-echo
    version: v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: http-echo
      version: v2
  template:
    metadata:
      labels:
        app: http-echo
        version: v2
    spec:
      containers:
        - name: http-echo
          image: http-echo:latest
          imagePullPolicy: IfNotPresent
          args: ["8080", "two"]
          ports:
            - containerPort: 8080

Apply the previous specification using the following command:

kubectl apply -f echo-service.yaml

We now have two services exposing the two versions of our application. However, we can’t access these services from outside the cluster. We will use Kong Ingress Gateway to expose our services and realize a few common use cases next.

Improve Traffic Management

Kong Istio Gateway can extend the traffic management patterns of Istio by adding capabilities such as GraphQL caching and routing to intermediary HTTP proxies. Implementing such complex policies is trivial with Kong. Let’s first expose the services through the gateway, after which we will implement a common traffic management use case.

Create two ingress objects that route traffic to the two application versions on the default endpoint and the /v2 endpoint. Doing so will look like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kong-ingress
  namespace: echo
spec:
  ingressClassName: kong
  rules:
    - http:
        paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: http-echo
                port:
                  number: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kong-ingress-2
  namespace: echo
spec:
  ingressClassName: kong
  rules:
    - http:
        paths:
          - path: /v2
            pathType: ImplementationSpecific
            backend:
              service:
                name: http-echo-2
                port:
                  number: 8080
---

Apply the specification to your cluster using the kubectl apply command. You can invoke the two endpoints from your browser and verify the output as follows:

Figure 3: Two versions of Http Echo application

Let’s now enforce rate limits on one of the versions of the application (v1). Let’s also write the specification for the rate limit plugin. This allows a single request to reach the service as follows:

apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: rate-limit
  namespace: echo
plugin: rate-limiting
config:
  minute: 1
  policy: local

Now, apply this policy by annotating the ingress object kong-ingress, which will enable Kong Gateway to apply the rate limit plugin on the routes covered by the ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    konghq.com/plugins: rate-limit
  name: kong-ingress
  namespace: echo
spec:
  ingressClassName: kong
  rules:
    - http:
        paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: http-echo
                port:
                  number: 8080

Hit refresh on your browser instances to see how this policy affects the behavior of your application:

Figure 4: Rate limit applied to a version of Http Echo application

Finally, let’s visualize our service mesh topology with Kiali. Kiali will also help us understand how the various services interact. Execute the following command to launch the Kiali UI:

istioctl dashboard kiali

Navigate to the Graph view of your application and enable the Traffic Animation option to visualize the current network traffic flow.

Figure 5: Kiali UI

You can explore a wide array of traffic control plugins available in Kong’s Plugin Hub. Iterating here can improve the traffic management capabilities of Istio.

Figure 6: Traffic control plugins in Kong Plugin Hub

Improve the Security of Istio Managed Services

Istio provides key security capabilities such as authentication, authorization, and certificate management. Together, these components ensure that only trusted traffic flows between the services in the mesh. Kong Istio Gateway can improve the security of the mesh to include capabilities such as bot protection and IP restriction.

Let’s configure the IP restriction plugin and apply it to version 2 (v2) of the HTTP Echo service as follows:

apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  name: ip-restriction
  namespace: echo
  annotations:
    kubernetes.io/ingress.class: kong
config:
  allow:
    - 54.13.21.1
plugin: ip-restriction
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    konghq.com/plugins: ip-restriction
  name: kong-ingress-2
  namespace: echo
spec:
  ingressClassName: kong
  rules:
    - http:
        paths:
          - path: /v2
            pathType: ImplementationSpecific
            backend:
              service:
                name: http-echo-2
                port:
                  number: 8080

After applying the policy, Kong Gateway will handle any request to the endpoint http://localhost/v2. It will respond with HTTP 403 as follows:

Figure 7: IP restriction applied to a version of Http Echo application

Improve Observability of Services

Istio has built-in support for distributed tracing, and since Kong Istio Gateway is yet another service in the service mesh, you will get end-to-end traces of requests traveling through the mesh. To view the traces generated by the application, let’s install Jaeger with the following command:

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.12/samples/addons/jaeger.yaml

Remove annotations from the ingress resources so that traffic will reach the Echo application without constraints. Send a few requests to any version of the service. Using the following command, generate some telemetry and launch the Jaeger dashboard:

istioctl dashboard jaeger

On the Jaeger dashboard, you can see the dependency graphs of the services as follows:

Figure 8: Dependency graph of the services

You can also track the traces generated by the services by filtering them in the search window. Distributed traces can help you identify performance issues in your application, and they are a great way to present the entire lifecycle of requests served by your microservices.

Figure 9: Distributed traces in Jaeger

The suite of Logging plugins from Kong Gateway can improve the observability of services in the mesh even further. For example, you can use plugins to channel the logs to Kafka and StatsD.

Figure 10: Logging plugins in Kong Plugin Hub

Let’s add logging capabilities to our services by attaching the http logging plugin to our ingress service. First, create a mockbin.org endpoint with default settings and then apply this configuration to the ingress service with the mockbin id inserted on the http_endpoint:

apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: http-log
  namespace: echo
config:
  http_endpoint: http://mockbin.org/bin/:id
  method: POST
  timeout: 1000
  keepalive: 1000
  flush_timeout: 2
  retry_count: 15
plugin: http-log
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    konghq.com/plugins: http-log
  name: kong-ingress
  namespace: echo
spec:
  ingressClassName: kong
  rules:
    - http:
        paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: http-echo
                port:
                  number: 8080

Send a few requests to the default endpoint and inspect the output in the bin that you created on https://mockbin.org/:

Figure 11: Logs transported to Mockbin by Kong logging plugin

Conclusion

This article discussed installing Kong Istio Gateway in the Istio service mesh as a regular service. We also discussed how the north-south and east-west traffic flows in the cluster and how the presence of Kong Istio Gateway aids with that traffic flow. Finally, using a few examples, we discussed how you could enhance the key capabilities of Istio using Kong Gateway. If you want to gain the power of Istio in your systems, give it a try yourself!

Minimizing Microservices Complexity with Reusable APIs

Rahul Rai — Tue, 28 Sep 2021 12:30:44 +0000

Developers build modern enterprise applications on the orchestration of a complex network of microservices. Over time, as the number of applications and microservices increases, the complexity of the orchestration increases as well.

Complexity in typical microservices architecture

Reusable APIs address the problem of complexity by allowing multiple applications to rely on a few independent microservices that expose the data and functions of their domain.

Why Should We Build Reusable APIs?

API sprawl is a living challenge in large organizations. It is not uncommon to hear from architects that their organizations have hundreds of APIs, many of which are unknown to teams in the company. The lack of visibility leads to teams creating even more redundant APIs, which adds to the existing problem.

The API sprawl problem depends on the scale, so creating duplicate APIs might not be a problem for small organizations with a small number of applications, services, and databases. On the other hand, bespoke API development might consume a lot of development time and effort for a large organization.

The challenges of API sprawl include:

Managing the APIs gets more complex as the number of APIs increases.
Introducing diverse databases that do not support interoperation.
Cross-cutting concerns and organizational policies such as security, logging, and monitoring are hard to implement consistently.

Since reusable APIs are not specific to a single application, an organization can have far fewer APIs in its portfolio. However, there are many benefits an organization can realize by introducing reusable APIs:

APIs can be easily cataloged and maintained.
Building a new application won't require creating new APIs. The reduced development cost will also reduce the time to market.
Reduced spend on security. Due to fewer connections between applications and web services, you can easily control and monitor access to your digital assets.
Applications and APIs can be scaled and migrated independently.

Adopting reusable APIs can be significantly challenging unless you have a comprehensive API platform that supports a low friction application of security, scalability, and monitoring best practices. Without an API platform, addressing common API concerns such as the following can become a challenge:

Applying policies on key areas such as security, logging, and caching uniformly across all APIs.
Consistently monitoring the APIs.
Maintaining documentation of the APIs.
Creating applications and connections on-demand to the APIs through a self-service platform used by application developers.

It is important to note that reusable APIs serve many applications simultaneously, and hence every reusable API is critical to the organization's operations. The API platform must be highly scalable and performant to avoid disruptions in operations.

One Solution to Help Developers Manage Reusable APIs: Kong

Kong is an open-source API platform that acts as middleware between the compute clients and APIs. The Kong platform contains a variety of plugins that allow you to extend the capabilities of APIs easily. In addition, operations teams and product owners can use the Kong platform to create self-service portals for developers interested in consuming the APIs, managing application and developer registrations without involving the API developers.

The Kong platform consists of the following three API proxy components (called gateway runtimes) used to manage and operate the upstream services:

Ingress Controller: Enables the application-to-application connectivity in a Kubernetes environment.
Kong Mesh: Enables connectivity between services of an application in Kubernetes or cloud environment.
Kong Gateway: Enables edge connectivity for services with clients.

Kong Konnect brings the gateway runtimes together on a single platform. Enterprises deploy their APIs on various hosts such as Kubernetes, cloud, and on-premise. Based on the host, you can choose the appropriate gateway runtime for your APIs and leverage the flexibility and scale of Kong to manage your services.

Regardless of the gateway runtime that you choose, you can use all the plugins and tools of the Kong ecosystem. The Konnect platform's core is the ServiceHub, which provides service catalog capabilities for the end-to-end lifecycle management of services.

Kong Konnect platform credit: Kong

Every Kong runtime is composed of two fundamental components:

Control plane: The control plane enables the operators to define routes, telemetry, and policies such as authentication, routing, and rate limiting.
Data plane: The data plane controls the exposure of the upstream APIs as per the policies defined in the control plane. The data plane is responsible for routing the network traffic, enforcing the policies, and emitting the telemetry.

Addressing the Horizontal Concerns of Microservices with Kong

The primary reason for adopting an API gateway such as Kong is to add abstractions. An API gateway can help you decouple the following:

External network from the internal network
External API interface from backend API interface
External API version from internal API version

Some broader API concerns that you can address with an API gateway include:

Enterprise policies: Operations teams can enforce certain policies without affecting the business APIs
- Authentication: Basic, API Key, LDAP, mTLS, OAuth/OIDC, and others
- Traffic control: Canary release, caching, rate limiting, and others
- Monitoring and analytics
- Logging: Datastream, file log, StatsD, and others
Security: Teams can place API gateways in the DMZ, and the security teams can focus on applying strict policies to mitigate the risks on the API gateways. Since the business APIs are not part of the DMZ, their security standards can be much lower. Also, the backend APIs run a low risk of receiving invalid messages because the API gateways inspect the messages in the DMZ and block illegal messages from propagating further.

Kong offers a wide range of plugins that security teams can use to address the security concerns of API gateways.

Kong security plugins

Request transformation: Kong supports transforming the requests and responses to enable heterogeneous services to communicate with one another. Kong can transform messages from one synchronous messaging protocol to another, such as gRPC to REST and REST to GraphQL. Kong can also transform synchronous requests to asynchronous requests with plugins such as the REST to Kafka messages transformer.

Kong transformation plugins

Deployment: You can deploy the Kong data plane to popular cloud or container platforms such as AWS, Azure, Heroku, and Kubernetes on any cloud.

What is ServiceHub, and What is its Role in APIOps?

The concept of APIOps is quite simple: bring DevOps and GitOps to the API and microservices lifecycle.

Let's discuss how the suite of Kong tools (Insomnia, inso, and decK) can help you implement APIOps for your microservices. Below are the typical stages of an APIOps deployment pipeline:

Design stage: Developers can use Insomnia to design API specs and write tests for their API. Insomnia will instantly lint your OpenAPI specs and validate them. You can also use Insomnia to specify Kong plugins and policies that need to be applied to your API endpoints for proper governance and security. Next, you can use Insomnia to generate the declarative config to set up a local Kong Gateway to verify the setup on your system.

Generate Kong declarative configuration

To push your API to the correct repository, you can leverage the built-in Git Sync feature.

CI stage: Once the new code is in the repository, the reviewer can use the Inso CLI and the build system to run automated tests and the OpenAPI specification linter. After validating the quality and governance of the API, Inso can generate a declarative config for Kong.
Deployment stage: The decK CLI can use the generated declarative config to adjust the state of the Kong runtime. decK maintains the state of the Kong Gateway runtime in a persistent store, and it can use the declarative config to detect the drift between the desired state and the gateway runtime. Once the drift is detected, decK can automatically bring the runtime to the desired state.

After the deployment is complete, the new API specs can be published to the Kong developer portal using the DevPortal CLI. The API developers can use the new API specs to create new applications without further coordination with the API team.

The following diagram illustrates these stages of an APIOps deployment pipeline:

APIOps pipeline credit: Kong

See it in Action: (Example) Pricing Service for an eCommerce Application

One common example of microservices is an eCommerce application that uses several services to power the customer experience.

Product pricing is one of the common concerns of several applications in the eCommerce ecosystem. Both the trading application for customers and the finance application for the internal finance team make use of the product pricing service. We can build a single, reusable product pricing service that fulfills the concerns of both applications. A single pricing service will ensure that the pricing information is always consistent and accurate. The development teams in the organization can build new products by using the same service, thereby reaching the market faster. Also, new pricing features such as discounts and taxes can be quickly added to the pricing service and made available to all the applications in the organization.

We will cover the steps to roll out the product pricing service to the developers in the organization in the following sections. First, we will deploy the pricing service in Kubernetes and then use the Kong Gateway runtime on Kubernetes as a proxy for the service.

Set Up a Kong Konnect Account

Sign up for a Kong Konnect account. You can start with the free tier and upgrade to the other pricing tiers based on your needs. Kong Konnect offers several resources to help you get started. You can follow their getting started documentation, blog post, or video to familiarize yourself with the platform.

Next, prepare a Kubernetes cluster in AWS, Azure, or GCP for deploying your service and the gateway runtime. For example, I created the following single-node Kubernetes cluster on Azure Kubernetes Service (AKS).

One node cluster in AKS

We will revisit our cluster soon. Let's first write the specs of the Prices API.

Write API Specs with Insomnia

Download and install Insomnia on your system. Click on the Design tab to launch the editor to write the OpenAPI specs for the pricing service.

Our API contains a single endpoint: /price/{product id} to fetch the price of a product. Write the following spec in the editor and inspect the preview in the split pane.

openapi: 3.0.0
info:
  description: Prices API is responsible for managing the latest prices of products.
  version: 1.0.0
  title: Prices API
paths:
  "/price/{id}":
    get:
      tags:
        - price
      summary: Find price by item id
      description: Returns the price of a single item
      operationId: getPriceById
      parameters:
        - name: id
          in: path
          description: ID of item
          required: true
          schema:
            type: integer
            format: int64
      responses:
        "200":
          description: successful operation
          content:
            application/xml:
              schema:
                $ref: "#/components/schemas/Price"
            application/json:
              schema:
                $ref: "#/components/schemas/Price"
        "400":
          description: Invalid ID supplied
        "404":
          description: Item not found
servers:
  - url: http://prices-api-service.pricing-ns.svc.cluster.local
components:
  schemas:
    Price:
      type: object
      properties:
        id:
          type: integer
          format: int64
          example: 1
        value:
          type: number
          format: decimal
          example: 30.5
        currency:
          type: string
          example: USD
        validUntil:
          type: string
          description: date till which the price is valid
          example: 2019-05-17
      xml:
        name: Price

OpenAPI spec of Prices API

You can use the built-in Git integration to share your work with your team. You can also debug and test your API inside Insomnia.

I have published the Docker image for the API based on the previous spec to Docker Hub. If you are interested, the following is the .NET implementation of the API, which is also available in my GitHub repository:

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

app.MapGet("/price/{id}", (int id) => new Price(id, id * 3.5, "USD", DateTime.UtcNow + TimeSpan.FromDays(5)));

app.Run();

record Price(int Id, double Value, string Currency, DateTime ValidUntil);

Use the following Kubernetes specification to deploy the pricing service to your cluster.

kind: Namespace
apiVersion: v1
metadata:
  name: pricing-ns
  labels:
    name: pricing
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prices-api-deployment
  namespace: pricing-ns
  labels:
    app: prices-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prices-api
  template:
    metadata:
      labels:
        app: prices-api
    spec:
      containers:
        - name: prices-api
          image: rahulrai/prices-api:latest
          ports:
            - containerPort: 8080
          resources:
            requests:
              memory: "64Mi"
              cpu: "250m"
            limits:
              memory: "128Mi"
              cpu: "500m"
---
apiVersion: v1
kind: Service
metadata:
  name: prices-api-service
  namespace: pricing-ns
spec:
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
      name: http
  selector:
    app: prices-api
--------

You can inspect the service and its ports by running the following command in a kubectl shell:

Inspect the Prices API service

Configure the Kong Runtime

Kong Konnect manages your control plane for you and allows you to use the Konnect portal or decK CLI to configure the required plugins that will help you add security and governance to your upstream service.

The data plane component, which will work according to the policies defined in the control plane, will be hosted in our cluster. First, we install the data plane component and connect it to the control plane.

Navigate to the Konnect portal and click on the Runtime option. Next, open the Kubernetes tab and click on the Generate Certificate button. Kong will generate the certificates that your data plane can use to establish a secure connection to the control plane. Copy the certificates and the unique Helm chart parameters files for your runtime.

Configurations for the data plane

Follow the instructions in the Kubernetes runtime configuration guide to set up a runtime with the certificates and configuration parameters that you previously recorded.

After installing the runtime, you can get the external IP address of the Kong Gateway by executing the following command:

kubectl get service my-kong-kong-proxy -n kong

External IP address of the Kong Gateway

Record the External IP address of the proxy, which is the access point for your data plane.

Once connected, you can view your runtime on the Runtimes page as follows:

Your connected Kong runtimes

We now have our service and the data plane running in the cluster. Also, our data plane is connected to the control plane. Next, we create a representation of the pricing service in the ServiceHub and create a route to enable the data plane to communicate with the Prices API service.

Creating a service in the ServiceHub

We need to link the gateway to the upstream service by creating an implementation of the service. First, click on the current version of the service, then click on the Add New Implementation button.

We specify the URL of the upstream service. The default URL of a service in Kubernetes follows the format: http://<service-name>.<namespace>.svc.cluster.local:<port>.

To extract the URL of any cluster service, execute the nslookup command in the shell of a pod as follows:

kubectl run temp-pod --rm -i --tty --restart=Never --image=radial/busyboxplus:curl -- nslookup prices-api-service.pricing-ns

Lookup the address of the Prices API service

Enter the service URL in the URL field of the form and click on the Next button.

Submit the URL of the Prices API service

Since an API gateway can interface multiple upstream services, you must specify the route path (or hostnames, headers, and so on) that the gateway can use to identify the requests destined for an upstream service. We will configure the route such that request to path /api/ will be routed to the service as follows:

Configure the route

Click on the Create button to create the route. The communication between the control plane and the data plane is fast. So, we can now send a request to the Prices API service via the gateway available at the external IP address that we previously recorded as follows:

curl http://<kong-external-ip>/api/price/<any-number>

Send a request to the Prices API service via Kong

You can view the traffic sent to the API in the Kong Vitals data. I simulated some successful requests and some errors to showcase how the data is displayed.

Kong vitals stats

Let's now make the Prices API available to the developers of our organization.

Publishing the Prices API to the Developer Portal

On the version details page, you will find the Version Spec card with the option to upload the API specification you created in Insomnia previously. The specification will help the developers understand the API request and response formats before building their applications on it.

From the service details page, you can upload Markdown-formatted documentation of your API for the developers to read.

Upload API documentation

To publish your service to the Dev Portal, click on the Publish to Portal option in the Actions menu.

Publish your service to the Dev Portal

To allow developers to use the service in their applications, click on the Enable app registration from the same menu.

When you enable app registration, an application will present its identity to the Kong Gateway to access the upstream service. You can choose between API Key based authentication (key-auth) and OAuth2 based authentication (openid-connect). For simplicity, choose the key-auth option. You can also enable the Auto approve toggle to automatically approve the developers' requests to build applications leveraging your service. Click on the Enable button to complete the process.

Enable app registration

Click on the Dev Portal option to view the list of published services in the Dev Portal and the link to the Dev Portal.

Dev Portal details

Visit the Dev Portal in a new tab and register for an account. Then, head back to the Konnect portal and click on the Connections and the Requests options to view the developer registration request. You can approve the request by clicking on the Approve button.

Approve the developer registration request

Use the newly approved developer account to access the self-service developer portal. You will find the Prices API in the list of services available to you for building applications. Click on the Prices API service card.

List of services available to developers

If you previously uploaded the API spec in the ServiceHub, you will find the UI for your API spec here. Developers can easily view the endpoints and understand the schema of request and response from this view.

Prices API specification

Click on the Register button to begin registering the API with an application. Since you don't have any applications, the following dialog will ask you to create a new application. To create a new application, click on the Create Application button.

Create a new application

Enter the details of the new application and click on the Create button.

Enter details of the new application

Finally, click on the Request Access button to request access to the API.

Request access to the API

Since, in the ServiceHub, we configured the service requests to be automatically approved, we can now use the service in our application without requiring further approvals.

We now need the identity of the application to access the API. Click on the Generate Credential button to generate a new credential for the application. It will instantly create an API key that you can use to access the upstream Prices API.

Generate an API key

Let's use this key to access the API as follows:

curl --request GET 'http://<gateway-ip>/api/price/50' --header 'apikey: <api-key>'

Access the API using the API key

To prevent consumers from sending an excessive number of requests to the API, let's add a plugin to limit the number of requests received by our API per minute.

Head back to the Konnect portal and navigate to the version view of the API in ServiceHub. Next, click on the New Plugin button in the Plugins card.

Add a new plugin to the gateway

Search for the Rate Limiting plugin and click on the Enable button.

Enable the Rate Limiting plugin

Configure the plugin to allow a maximum of two requests per minute by setting the value of the property Config.Minute to 2. Also, set the value of the property config.policy to local to enable the gateway to persist the state in the local memory.

Configure the Rate Limiting plugin

Click on the Create button to add the plugin to the list of plugins active on the service version.

Try sending multiple requests to the API and observe the response. You will find that the gateway starts throttling the requests after the second request:

Rate limiting plugin in action

You can add more plugins to enforce other policies on your APIs. You can find the list of available plugins and their documentation on the Kong Plugin Hub.

Conclusion

Adopting the principle of using reusable APIs can help an organization eliminate the waste of precious resources by reusing the same API for multiple applications. In addition, reusable APIs can make your microservices ecosystem easier to maintain while improving the velocity of your application development at the same time.

The Kong Konnect platform and its family of tools can help you implement APIOps and automate the delivery and management of reusable APIs. Kong Konnect enforces a clear separation between operations and application developers through the ServiceHub and the Developer Portal.

On the one hand, the ServiceHub allows the operations to onboard APIs and manage their versions. On the other hand, the Dev Portal enables the application developers to be autonomous in building new applications.

Implementing cross-cutting concerns of APIs such as authentication and caching requires a thorough understanding of the best practices and time and effort in implementation. The Kong Plugin Hub allows the operations team to enforce organization-specific policies across the APIs without wasting time and money.

With Kong Konnect, the burden of implementing non-functional requirements regarding a business API is no longer the responsibility of the developers. With a clear separation of responsibility, each team can concentrate on its strengths.