Forem: k6

Get Started with k6 browser

Marie Cruz — Wed, 11 Jan 2023 11:28:57 +0000

This post explains how to get started with k6 browser, a k6 experimental module that adds browser-level APIs to interact with browsers and collect web performance metrics as part of your k6 tests.

Testing Beyond Protocol Level

Over the years, k6 has become known as a performance testing tool that provides the best developer experience. Most of our efforts have focused on providing a tool that helps test your servers or backend systems. Our comprehensive load testing guide recommends keeping users in mind, and backend performance testing only addresses half of your performance testing efforts.

Suppose you test the user experience of your website and verify that there are no performance issues on a specific user journey. In that case, you need to drive some of your performance testing efforts from a browser perspective and consider a more realistic end-to-end test of the user flow.

Most load-testing tools focus on testing API endpoints, but it's different from what your users normally interact with. Your users interact with the browser, so it's also vital to test the browser's performance to get an end-to-end perspective of what's happening when interacting with your web applications.

Both frontend and backend performance testing has its pros and cons when done in isolation, which we discussed in more detail as part of the video below.

Here at k6, we want to start expanding our performance testing use case and also test beyond the protocol level.

This is where the k6 browser module comes in.

What is k6 browser module?

The browser module brings browser automation and end-to-end web testing to k6 while supporting core k6 features. It enables you to get insights from your front-end application during performance testing.

You can now mix browser-level and protocol-level tests in a single and unified script using k6. This can simulate the bulk of your traffic from your protocol-level tests and run one or two virtual users on a browser level to mimic how a user interacts with your website, thus leveraging a hybrid approach to performance testing.

The browser module offers a unique solution as you don’t have to use separate tools to test your frontend and backend systems. It also offers a simplified experience and aggregated view of performance metrics.

Writing the test

You can copy one of our example scripts to get started as part of our k6 browser module documentation.

import { chromium } from 'k6/experimental/browser';

export default function () {
  const browser = chromium.launch({ headless: false });
  const page = browser.newPage();
}

Let’s break down what’s happening with the preceding code.

We are importing chromium from the k6/experimental/browser module. chromium is of type BrowserType, which is k6 browser’s entry point into launching a browser process.
Next, we use the scenario function, an existing k6 functionality, to define our VU (virtual user) code.
To create a new browser instance, we use the launch method of chromium, which returns a Browser object. You can pass different parameters within launch and one of the parameters you can pass is headless, which you can use to show the browser or not.
To create a new page in your browser instance, we use the browser.newPage().

Now, on to the fun part! Let’s simulate a user visiting a test application and logging in.

import { chromium } from 'k6/experimental/browser';
import { check } from 'k6'

export default async function () {
  const browser = chromium.launch({ headless: false });
  const page = browser.newPage();

  try {
    await page.goto('https://test.k6.io/my_messages.php', { waitUntil: 'networkidle' });

    page.locator('input[name="login"]').type('admin');
    page.locator('input[name="password"]').type('123');

    await Promise.all([
      page.waitForNavigation(),
      page.locator('input[type="submit"]').click(),
    ]);

    check(page, {
      'header': page.locator('h2').textContent() == 'Welcome, admin!',
    });
  } finally {
    page.close();
    browser.close();
  }
}

There are a lot of things happening in the preceding code especially the introduction of asynchronous operations so let’s break it down again.

We visit the page by using page.goto and pass the test application URL. We are also waiting for the network to be idle, which will succeed if there are no network connections for at least 500 ms. page.goto is also an asynchronous operation, so we need to wait for this to finish and use the await keyword.
Once the operation completes, we use page.locator to interact with the elements we want. In the example, we are creating two locators. One for the login name and another one for the login password. We use the type method to type the name and password into the fields.
To click the login button, we use the click method, which is an asynchronous operation. Clicking the submit button also causes page navigation which we need to wait to load, so page.waitForNavigation(), another asynchronous operation, is needed because the page won't be ready until the navigation completes.
Since there are two asynchronous operations, we need to use Promise.all([]) to wait for the two promises to be resolved before continuing to avoid any race conditions.
Next, we use the check feature from k6 to assert the text content of a specific element.
Finally, we close the page and the browser.

Running the Test

To run the test, simply use the following command.

K6_BROWSER_ENABLED=true k6 run script.js

If you face any issues running the command, please check out our documentation for running the test.

You should see a similar test run as this video.

With the browser launching, this provides a more visual experience as to what your users actually see so you can also find blind spots and catch issues related to browsers that won't be detected on a protocol-level.

Browser Metrics

When it's finished running, apart from the metrics that k6 already tracks, additional browser metrics are tracked as part of the k6 summary output.

  browser_dom_content_loaded.......: avg=36.72ms min=544µs   med=22.61ms max=87.02ms  p(90)=74.14ms  p(95)=80.58ms
  browser_first_contentful_paint...: avg=47.52ms min=22.01ms med=47.52ms max=73.02ms  p(90)=67.92ms  p(95)=70.47ms
  browser_first_meaningful_paint...: avg=75.22ms min=75.22ms med=75.22ms max=75.22ms  p(90)=75.22ms  p(95)=75.22ms
  browser_first_paint..............: avg=45.72ms min=21.96ms med=45.72ms max=69.49ms  p(90)=64.73ms  p(95)=67.11ms
  browser_loaded...................: avg=38.14ms min=5.28ms  med=22.45ms max=86.68ms  p(90)=73.83ms  p(95)=80.26ms

The metrics above are still subject to change. Our goal is to be able to report the Core Web Vitals as well as Other Web Vitals for a better understanding of user experience.

k6 Browser API

The k6 Browser API aims to provide a rough compatibility with the Playwright API for NodeJS, meaning k6 users don't have to learn an entirely new API.

At the moment, the k6 API is synchronous. However, since many browser operations happen asynchronously, and in order to follow the Playwright API more closely, we are working on migrating most of the browser methods to be asynchronous. This means that while k6 browser is ready to be used, be warned that our API is still undergoing a few changes.

At the moment, few methods such as page.goto(), page.waitForNavigation() and Locator.click() return JavaScript promises. Our goal is to support async and await syntax for all asynchronous operations, for simplicity.

For more examples on how to use the k6 browser API, please check out k6 browser examples.

A Hybrid Approach to Performance Testing

If you only consider web performance, this can lead to false confidence in overall application performance when the amount of traffic against an application increases.

It's still highly recommended to also test your backend systems to have a complete picture of your application’s performance, via the protocol-level.

However, there are problems associated with testing via the protocol level, such as:

not being closer to the user experience since it’s skipping the browser,
scripts can be lengthy to create and difficult to maintain as the application grows,
browser performance metrics are ignored.

On the other hand, if you perform load testing by spinning up a lot of browsers, this requires significantly more load-generation resources, which can end up quite costly.

To address the shortcomings of each approach, a recommended practice is to adopt a hybrid approach to performance testing, which is a combination of testing both the backend and frontend systems via protocol and browser level. With a hybrid approach, you spin up the majority of your load via the protocol level, then simultaneously have one or two browser-level virtual users, so you can also have a view of what’s happening on the front end.

The great thing with k6 browser module is that it can offer you this hybrid approach to performance testing. While you can do this with multiple tools, the beauty of using this module is that it's built on top of k6, which means that you can have a protocol-level and a browser-level test in the same script.

Let's see how that translates to code.

Writing the test

A common scenario that we recommend is to mix a smaller subset of browser-level tests with a larger protocol-level test. To run a browser-level and protocol-level test concurrently, you can use scenarios.

import { chromium } from 'k6/experimental/browser';
import { check } from 'k6';
import http from 'k6/http';

export const options = {
  scenarios: {
    browser: {
      executor: 'constant-vus',
      exec: 'browser',
      vus: 1,
      duration: '10s',
    },
    news: {
      executor: 'constant-vus',
      exec: 'news',
      vus: 20,
      duration: '1m',
    },
  },
};

export async function browser() {
  const browser = chromium.launch({ headless: false });
  const page = browser.newPage();

  try {
    await page.goto('https://test.k6.io/browser.php', { waitUntil: 'networkidle' });

    page.locator('#checkbox1').check();

    check(page, {
      'checkbox is checked': page.locator('#checkbox-info-display').textContent() === 'Thanks for checking the box',
    });
  } finally {
    page.close();
    browser.close();
  }
}

export function news() {
  const res = http.get('https://test.k6.io/news.php');

  check(res, {
    'status is 200': (r) => r.status === 200,
  });
}

Let's break down the preceeding code again.

We are using options to configure our test-run behaviour. In this particular script, we are declaring two scenarios to configure specific workload, one for the browser-level test called browser and one for the protocol-level test called news.
Both the browser and news scenario are using the constant-vu executor which introduces a constant number of virtual users to execute as many iterations as possible for a specified amount of time.
Next, there are two JavaScript functions declared, browser() and news(). These functions contain the code that will be executed by a virtual user. The browser() function, represents our browser-level test and simply visits a test URL, clicks a checkbox and verifies if the checkbox has been ticked successfully while the news() function, which represents our protocol-level test, sends a GET request to a different URL and checks if the status code is returning 200.
Since we are using scenarios, the two functions are independent from each other and therefore, runs in parallel.

Running the test

Using the same k6 run command as above, you should see a similar test output as below:

running (1m00.1s), 00/21 VUs, 12953 complete and 0 interrupted iterations
browser ✓ [======================================] 1 VUs   10s
news    ✓ [======================================] 20 VUs  1m0s

     ✓ status is 200
     ✓ checkbox is checked

     browser_dom_content_loaded.......: avg=12.27ms  min=68µs    med=11.76ms max=26.77ms  p(90)=25.56ms p(95)=26.36ms
     browser_first_contentful_paint...: avg=21.93ms  min=12.5ms  med=25.32ms max=26.19ms  p(90)=25.83ms p(95)=26.01ms
     browser_first_paint..............: avg=21.88ms  min=12.45ms med=25.27ms max=26.14ms  p(90)=25.78ms p(95)=25.96ms
     browser_loaded...................: avg=12.18ms  min=984µs   med=11.74ms max=25.65ms  p(90)=24.37ms p(95)=25.19ms
     checks...........................: 100.00% ✓ 12953      ✗ 0
     data_received....................: 21 MB   341 kB/s
     data_sent........................: 1.4 MB  24 kB/s
     http_req_blocked.................: avg=2.14ms   min=1µs     med=2µs     max=290.37ms p(90)=4µs     p(95)=4µs
     http_req_connecting..............: avg=1.09ms   min=0s      med=0s      max=195ms    p(90)=0s      p(95)=0s
     http_req_duration................: avg=90.59ms  min=80.93ms med=92.8ms  max=542.92ms p(90)=96.89ms p(95)=102.83ms
       { expected_response:true }.....: avg=90.49ms  min=80.93ms med=92.8ms  max=542.92ms p(90)=96.86ms p(95)=102.81ms
     http_req_failed..................: 0.00%   ✓ 0          ✗ 12946
     http_req_receiving...............: avg=154.41µs min=17µs    med=47µs    max=97ms     p(90)=67µs    p(95)=76µs
     http_req_sending.................: avg=20.36µs  min=0s      med=14µs    max=32.36ms  p(90)=20µs    p(95)=21µs
     http_req_tls_handshaking.........: avg=1.16ms   min=0s      med=0s      max=183.1ms  p(90)=0s      p(95)=0s
     http_req_waiting.................: avg=90.41ms  min=80.85ms med=92.72ms max=542.86ms p(90)=96.77ms p(95)=102.74ms
     http_reqs........................: 12960   215.658713/s
     iteration_duration...............: avg=93.58ms  min=81.05ms med=92.95ms max=1.83s    p(90)=97.54ms p(95)=103.37ms
     iterations.......................: 12953   215.542231/s
     vus..............................: 20      min=20       max=21
     vus_max..........................: 21      min=21       max=21

Since it's all in one script, this allows for greater collaboration amongst teams and a unified view of the performance metrics from a browser-level and protocol-level perspective.

Get Involved

While k6 browser started off as an extension, as of k6 version 0.43.0, it is now bundled in k6 as an experimental module, and usable without a separate binary or compilation step! We also have further plans to integrate k6 browser in k6 cloud as part of a private beta. We consider browser automation an important part of web application testing, and we have big goals for k6 browser. Our roadmap details essential status updates and our short, mid, and long-term goals.

With that said, we need your help! Since k6 browser is still relatively new, we need help from the community to try out the tool and give us feedback.

Check our GitHub project, read our documentation, and play with the tool. If you find any issues, please raise them on our GitHub project or check out our community forum for additional support.

Testing the new AWS ARM CPU's with k6

Carlos Ruiz Lantero — Thu, 25 Nov 2021 11:17:55 +0000

Recently AWS made the new ARM processors for Lambda functions generally available. With that change Serverless functions now use Graviton2, said to offer better performance at lower cost.

I built a sample API on AWS using API Gateway and Lambda, and I wrote two endpoints, one CPU-intensive (calculating Pi using Leibniz's formula), the other a typical data transfer endpoint (returning an arbitrary number of bytes). Two very different endpoints for my experiment.

One hard limit on the size of my experiment was that my personal AWS account allows me no more than 50 unreserved concurrency, which roughly means I cannot have more than 50 Lambda instances running in parallel. However, you don't need big numbers in order to gain insights.

The tests

I deployed both endpoints in both x86_64 (Intel) and arm64 (Graviton2). The tests were largely the same, ramp up type tests. For the CPU-intensive one I used a large number of iterations of the Leibniz formula (half a million to be precise). The data tranfer one requested 100 kilobytes.

The CPU-intensive test:

// Parts omitted

export const options = {
    scenarios: {
        x86: {
            // Parts omitted
            env: {
                ARCH: 'x86',
                ITERATIONS: '500000'
            }

        },
        arm: {
            // Parts omitted
            env: {
                ARCH: 'arm',
                ITERATIONS: '500000'
            }
        }
    }
}

export default function () {
    // Parts omitted
    url.pathname = `/pi/${__ENV.ARCH}`;
    url.searchParams.append('iterations', ENV.ITERATIONS);
    // Parts omitted
}

And the data transfer test:

// Parts omitted

export const options = {
    scenarios: {
        x86: {
            // Parts omitted
            env: {
                ARCH: 'x86',
                COUNT: '102400'
            }

        },
        arm: {
            // Parts omitted
            env: {
                ARCH: 'arm',
                COUNT: '102400'
            }
        }
    }
}

export default function () {
    // Parts omitted
    url.pathname = `/zerobytes/${__ENV.ARCH}`;
    url.searchParams.append('count', ENV.COUNT);
    // Parts omitted
}

Results (data transfer)

The test results for the data tranfer endpoint were roughly what I had expected. Little difference between the two architectures, seeing that the endpoint doesn't really push the CPU to do much.

The one interesting thing to note though is that the response time starts at 250ms before stabilizing at 100ms. This is typical for Lambda cold start, Lambda cold start lays within the ~100/200ms range. When you first trigger a Lambda function, AWS needs to spin up the infrastructure underneath, likely a container. Once you have your instances running, all your subsequent requests take much less time.

Results (CPU load)

For our CPU-intensive endpoint again we see that cold start, although the graph isn't as steep as in the previous example, due to regular response time being much bigger in comparison. What's interesting here though is that x86 is 20% faster than the new chips, which is exactly the opposite of what AWS claims.

Results

What I learned from this experience is that while AWS boldly claims arm is 20% faster and 20% cheaper, the results of my experiment say it's rather 20% slower and 20% cheaper. This might vary based on the chip instructions you use, this implementation of the Leibniz formula consists of loops, assigments, additions and other basic mathematical operations. But it turns out others are seeing similar results, like the folks at Narakeet.

That's not bad news though. Non-CPU workloads still benefit from the price at no speed trade-off. Arm also draws less power, which is better for the planet.

While the experiment covers Lambda, these findings could be extrapolated to EC2.

The concurrency (number of lambda instances running at the same time) roughly equals the number of requests per second you're running for your API, multiplied by the response time in seconds. It'd be interesting to monitor how this behaves over time and with higher load.

Finding .NET Memory Leaks through Soak Testing

Carlos Schults — Tue, 02 Nov 2021 15:25:57 +0000

As you’re probably aware, C# is a modern, garbage collected language, which means you don’t have to take care of object disposal as you would in languages like C++. However, that doesn’t mean .NET engineers don’t face memory problems, such as memory leaks.

In this post, we’ll walk you through the basics of memory leaks in .NET: what they are, how they occur, and why they matter. We’ll also show you how it’s possible to use techniques and tools already at your disposal — in particular, soak testing — to diagnose and fix memory issues.

Let’s get started.

.NET Garbage Collection 101

To start, we’ll briefly explain how the .NET garbage collection process works. To learn about GC in more detail, refer to Microsoft’s Fundamentals of garbage collection.

An Intro To GC

At a high level, GC works like this.

The analysis starts with an object that’s guaranteed to be alive, which are called “root objects.” For instance, local variables.
From the root object, the GC starts to follow all of its references until it reaches the end of the graph. Every visited object is marked as alive.
The process starts again for the other root objects.
By the end of the process, unvisited objects are deemed unreachable and marked for deletion.
Dead objects are cleaned from memory, but that leaves a lot of “holes” in memory. It’s better if the free areas in memory are kept together, so memory is compacted after the cleaning process.

What we’ve just described is an oversimplified view of GC; we left a lot of details out on purpose. However, you can already see that the process is a lot of work. It takes a while to complete, and thread execution is halted during the process.

Thus, garbage collection, despite being valuable, is costly for your app’s performance. How to get out of this conundrum?

Talking ‘Bout My Generation: Making GC More Efficient

To solve this dilemma, we need to run the collection process as infrequently as we can get away with it. Since every time counts, we’ve got to make sure GC is as efficient as possible.
.NET achieves this by using an approach that understands that not all objects are created equal: it organizes them in different spaces in memory called “generations.”

Generation 0 (G0) is the first generation, where newly created objects go by default. This generation is the one where collection happens more often, based on the assumption that young objects are less likely to have relationships with other objects and are more likely to be eligible for collection as a result.

Generation 1 (G1) is the next generation. Objects from G0 who survive a collection are promoted to G1. Here, collection happens less frequently.

Finally, you have Generation 2 (G2), which is the generation for long lived objects. Here, collection happens very rarely. Objects that make it to this generation are likely to be alive for as long as the application is in use.

GC operates on some assumptions. One of them is that large objects live longer.

If a large object were to be allocated in G0, that would create performance problems. It’d have to progress until G2, and the process of compacting it would be costly.

But that’s not what happens. Large objects (> 85 KB in size) are automatically promoted to an area called the Large Object Heap (LOH.) Unlike the Small Object Heap (SOH), the LOH doesn’t have generations, it’s collected along with G2.

.NET Memory Leaks 101

Having covered the fundamentals of the garbage collection process, let’s move on to memory leaks.

What Are Memory Leaks In .NET and Why Do They Happen?

Wikipedia defines memory leaks as follows:

[...] a memory leak is a type of resource leak that occurs when a computer program incorrectly manages memory allocations in a way that memory which is no longer needed is not released. A memory leak may also happen when an object is stored in memory but cannot be accessed by the running code.

It might seem counterintuitive that .NET can suffer from memory leaks from the definition above. After all, aren’t we talking about a managed environment?

Michael Shpilt argues that are two types of memory leaks in .NET:

managed memory leaks, which happen when you have obsolete objects that don’t get collected.
unmanaged memory leaks, which happen if you allocate unmanaged memory and don’t free it.

Why Should You Care About Memory Leaks?

Memory leaks can give you a severe headache if you don’t have them on your radar.

Suppose you have a number of short-lived objects which are no longer needed, but they can’t be collected, because they’re being referenced by other objects. The fact that they’re unnecessarily kept alive will impact performance: besides the unnecessary memory they use, they will cause collections, which are costly.

Besides degrading performance, severe memory leaks can cause an OutOfMemory exception which will crash your application.

Seeing a Memory Leak In Action

Now, I’ll use soak testing to allow us to see a memory leak in action. This will be a simulated example; I’ll use an application I deliberately modified to introduce a memory problem (described in this blog post).

You can download the modified version from this GitHub repo. Clone it using Git or download everything as a .zip file.

[HttpGet]
        [Route("{id:int}", Name = nameof(GetSingleFood))]
        public ActionResult GetSingleFood(ApiVersion version, int id)
        {
            strings.Add(new String('x', 10 * 1024)); // this is the offending line
            FoodEntity foodItem = _foodRepository.GetSingle(id);

strings in the code above refer to a static property of type ConcurrentBag. To sum it up:

with each request to this particular endpoint—which is the endpoint we’ll be testing during our endurance test—I create a useless big string.
the created string is added to a static collection, which ensures its reference will live on and not be collected.

So, with just two lines of code we ensure the allocation of unnecessary memory and its artificially induced longevity.

Now, I’ll go to my terminal, access the project’s folder and run the following commands:

cd SampleWebApiAspNetCore
dotnet run --configuration Release

After that, I can open my browser at https://localhost:5001/API/v1/Foods to see the API returning an array of foods:

// 
//

{
    "value": [
        {
            "id": 2,
            "name": "Hamburger",
            "type": "Main",
            "calories": 1100,
            "created": "2021-08-24T08:28:57.6607216-03:00",
            "links": [
                {
                    "href": "https://localhost:5001/api/v1/foods/2",
                    "rel":"self",
                    "method": "GET"
                },
                {
                    "href": "https://localhost:5001/api/v1/foods/2",
                    "rel":"delete_food",
                    "method": "DELETE"
                },
                {
                    "href": "https://localhost:5001/api/v1/foods/2",
                    "rel":"create_food",
                    "method": "POST"
                }
            ],
        }
    ]
}

Running Soak Testing With k6

Earlier, I’ve mentioned there are techniques and tools you can use to fight memory problems. It’s now time to meet the technique (soak testing) and the tool (k6) I’ll use in this tutorial.

What is Soak Testing?

There are many kinds of performance testing, and soak testing is one of them.

Unlike load testing, which is targeted at evaluating the app’s performance at a given point in time, soak testing determines how reliable the application is over an extended period of time. That’s why soak testing is also called endurance testing.

How does soak testing work in practice? You basically put a lot of pressure on your system for a few hours. That way, you simulate what would’ve been days of traffic in way less time, potentially uncovering memory leaks, poor configuration, bugs or edge cases that would only surface after an extended period of time.

Getting Started With k6

For our tests, we’ll use k6, which is an open-source tool for scriptable performance testing.

If you want to follow along, please refer to the installation guide. To test your installation, try to follow the minimum test example from k6’s documentation.

Creating The Test Script

k6 is a scripted tool. That means you write a script that details the steps that will be taken during the test. In the case of k6, you write scripts in JavaScript, which is great: JavaScript is the language of the web, so it’s very likely you and your coworkers have at least some familiarity with it.

I created a file called soak.js, with the following content:

import http from 'k6/http';
import { sleep } from 'k6';

export let options = {
  stages: [
    { duration: '2m', target: 200 },
    { duration: '56m', target: 200 },
    { duration: '2m', target: 0 },
  ],
};

const API_BASE_URL = 'https://localhost:5001/API/v1/';

export default function () {
  http.batch([
    ['GET', `${API_BASE_URL}Foods/1/`],
    ['GET', `${API_BASE_URL}Foods/2/`],
    ['GET', `${API_BASE_URL}Foods/3/`],
    ['GET', `${API_BASE_URL}Foods/4/`],
  ]);

  sleep(1);
}

Now, the explanations:

The script starts with the necessary imports;
Then, it declares an object named options, that represents the configuration for the test, detailing three different stages:
- the first will last for 2 minutes, ramping from 0 to 200 VUs (virtual users)
- the second will last for 56 minutes, staying at 200 users
- the last one will go down to zero users again, lasting for 2 minutes
After that, we declare the URL for the calls, and the endpoints of the API that will be hit during the test.

As you can see, with k6 you only need a few lines of code to create a test that will put the application under a heavy pressure for an extended period of time. More specifically, it’s this line:

{ duration: '56m', target: 200 },

that’s responsible for making this test a “soak”, or endurance test. It ensures k6 will hit the API simulating 200 users for almost an hour.

Of course, the configuration we’re using is just an example. In real usage scenarios, you implement a k6 script simulating a realistic user flow with the expected frequent traffic.

Running The Test Script

Now, I’ll run the test. I go to my terminal and run this:

k6 run soak.js --out cloud

Before this, I logged in my k6 Cloud account using my account token. The --out cloud option means that the test itself is run locally, but the results get sent to k6 Cloud, where I can analyze them afterward.

Analyzing The Results

The image above displays the results of the first test. Here’s the overview for the test run:

514676 requests were made.
From those, 37511 resulted in failure (or 7.29%)—these are tracked by the red line.
The average request rate was 687 requests per second, peaking at 797.71—this value is tracked in the chart with the purple line.
The average response time was 34 milliseconds, tracked with the sky blue line.

From the graph above, we can see that, after some 13 minutes of testing, the application simply couldn’t handle the pressure and crashed, and after that point, all requests resulted in failure until the end of the test run.

Now, take a look at the following image:

This is a screenshot from Visual Studio’s profiler. As you can see, even before reaching the 2 minute mark, the program’s memory consumption skyrocketed and two garbage collections had already taken place!

Changing The Application’s Code

“Fixing” the memory issue of this application would be the easiest task ever, since I was the one who introduced it in the first place! Here’s what the correct version of the method should look like:

[HttpGet]
        [Route("{id:int}", Name = nameof(GetSingleFood))]
        public ActionResult GetSingleFood(ApiVersion version, int id)
        {
            FoodEntity foodItem = _foodRepository.GetSingle(id);

            if (foodItem == null)
            {
                return NotFound();
            }

            return Ok(ExpandSingleFoodItem(foodItem, version));
        }

Were this a real situation, though, how could the results from this test run help us in detecting the underlying issue?

Since we’re testing the Foods endpoint on this API, a good place to start the investigation would be the controller action responsible for that specific GET endpoint.

In our example, we would quickly find the code responsible for creating a lot of memory leakage. In a real scenario, that would be the starting point of a debugging session. With the help of a profiler, you’d be more likely to find the culprit.

Running Soak Testing With k6, Again

After “fixing” the code, I’ll now run the tests again, so we can compare the results of both test runs.

Here are the results:

This time 2473732 requests were made.
From those, only 764 resulted in failure, which corresponds to 0.03%, as you can see in the red line in the chart.
This time, the purple line, tracking the successful requests, remains stable for virtually the entire session, just taking a deep dive about midway through the tests, but then making a full recovery.
At the same spot, you can see the request time—tracked by the sky blue line—skyrocketed.
The average request rate was 143 requests per second, peaking at 800.
The average response time was 21 milliseconds.

Conclusion

.NET isn’t immune to memory issues, among which memory leaks are probably the most insidious ones. Memory leaks are often silent; if you don’t watch out for them, they can sneak in your application and corrode its performance until it’s too late.

The good news is that it’s possible to fight back. By understanding the fundamentals of GC and memory management, you can write code in such a way that minimizes the likelihood of problems.

Education can only take you so far, though. At some point, you have to leverage the tools and techniques available to you.That includes profilers, such as CodeTrack.

Another great tool for your arsenal is k6. Although more widely known as a load testing tool, k6 also enables you to perform soak/endurance testing to your applications, which can help you uncover memory issues.

How to Load Test Your Kafka Producers and Consumers using k6

Ng Wai Foong — Wed, 22 Sep 2021 01:34:25 +0000

Recently, k6 started supporting k6 extensions to extend k6 capabilities for other cases required by the community. The community has already built plenty of extensions. k6 extensions are written in Go, and many of them are reusing existing Go libraries.

This makes k6 to be a versatile tool to test different protocols and adapt to multiple cases. This post is the third part of my series of articles testing various systems using k6:

Let's look in this post how we test the popular Kafka project. Apache Kafka is a powerful event streaming platform that provides the following features:

Write and read streams of events
Store streams of events for as long as you want to
Process streams of events in parallel retrospectively

It works by having client applications write events to the Kafka server. We term this type of application as Producers. Client applications which read and process events from the Kafka server are called Consumers.

Kafka itself is capable of handling hundreds to millions of events per second seamlessly on simple setup. But what if you wanted to test and observe how your Kafka service behaves before going live?

The xk6-kafka extension provides some convenient functions for interacting with Kafka Producers and Consumers. It serves as a producer that can send a high volume of messages per second, allowing you to monitor the system under test (SUT) and test how the application will keep up with the load.

xk6-kafka

At the time of this writing, the xk6-kafka extension provides the following APIs:

Function	Description
consume(reader, limit)	Consume messages from the Kafka server.
createTopic(address, topic)	Create a new topic.
listTopics(address)	Return a unique set of topics.
produce(writer, messages)	Produce messages to the Kafka server.
reader(brokers, topic)	Instantiate a new Reader instance.
writer(brokers, topic)	Instantiate a new Writer instance.

Some of the APIs mentioned above do accept additional optional parameters meant for authentication and message compression. Refer to more examples for additional information.

Building k6 with the kafka extension

By default, k6 does not support testing Kafka. Building k6 with the xk6-kafka extension creates a k6 version with the capabilities to test Kafka producers and consumers.

Make sure you have the following installed and ready before you continue:

Go (>=1.7)
Git

Next, continue the installation by running the following command in your terminal to install the xk6 module:

go install go.k6.io/xk6/cmd/xk6@latest

Once the command finishes successfully, you can start making your own custom k6 binary for Kafka as follows:

xk6 build --with github.com/mostafa/xk6-kafka@latest

It will take some time for the process to create a new k6 binary in your working directory.

Troubleshooting: If you experienced issues installing xk6 extension, kindly head over to the Github repository to download the pre-compiled binaries instead.

Running Kafka

The recommended approach is to use docker as manual installation is quite complicated and prone to errors. You can pull the following image by lensesio from DockerHub. It contains the complete Kafka setup for development.

docker pull lensesio/fast-data-dev:latest

After that, run the following to start the docker in detached mode:

sudo docker run -d --rm --name lenseio -p 2181:2181 -p 3030:3030 \
       -p 8081-8083:8081-8083 -p 9581-9585:9581-9585 -p 9092:9092  \
       -e ADV_HOST=127.0.0.1 lensesio/fast-data-dev

sudo docker logs -f -t lenseio

Visit http://localhost:3030 to get into the fast-data-dev environment.

k6 test

Import

Now, let’s create a new JavaScript file called test_script.js in the same directory as your k6 binary. Then, add the following import statement at the top of the file:

import { check } from "k6";
import { writer, produce, reader, consume, createTopic } from "k6/x/kafka";

Initialization

Continue by appending the following initialization code:

const bootstrapServers = ["localhost:9092"];
const kafkaTopic = "xk6_kafka_json_topic";

const producer = writer(bootstrapServers, kafkaTopic);
const consumer = reader(bootstrapServers, kafkaTopic);

The code will initialize both the writer and reader instances based on the configuration specified. If you are using a different IP/host address and port for your Kafka server, kindly modify it accordingly.

Next, call the createTopic function to create a new topic. Rest assured that this function will do nothing if the topic already exists.

createTopic(bootstrapServers[0], kafkaTopic);

Let’s create a function which generates a random integer as a unique identifier for each message later on. Please be noted that this is optional and not a mandatory requirement to do load testing.

function getRandomInt(max=1000) {
  return Math.floor((Math.random() * max) + 1);
}

Default function

As for the default function, define it as follows:

export default function () {
    let messages = [{
      key: JSON.stringify({
          correlationId: "test-id-sql-" + getRandomInt(),
      }),
      value: JSON.stringify({
          title: "Load Testing SQL Databases with k6",
          url: "https://k6.io/blog/load-testing-sql-databases-with-k6/",
          locale: "en"
      }),
    },
    {
      key: JSON.stringify({
          correlationId: "test-id-redis-" + getRandomInt(),
      }),
      value: JSON.stringify({
          title: "Benchmarking Redis with k6",
          url: "https://k6.io/blog/benchmarking-redis-with-k6/",
          locale: "en"
      }),
  }];

    let error = produce(producer, messages);
    check(error, {
          "is sent": (err) => err == undefined,
    });
}

The code block above works as follows:

Initialize a list of messages
Call produce function to publish the messages
Check if messages are successfully sent

Teardown

Once you are done with it, create a teardown function and close the connections:

export function teardown(data) {
    producer.close();
    consumer.close();
}

Run the test

Save the file and run the following command on your terminal:

./k6 run --vus 50 --duration 5s test_script.js

You should see the following output:

running (05.0s), 00/50 VUs, 15136 complete and 0 interrupted iterations
default ✓ [======================================] 50 VUs  5s

    ✓ is sent

    █ teardown

    checks.........................: 100.00% ✓ 15136  ✗ 0
    data_received..................: 0 B    0 B/s
    data_sent......................: 0 B    0 B/s
    iteration_duration.............: avg=16.49ms min=31.9µs med=13.52ms max=1.14s p(90)=28.55ms p(95)=36.46ms
    iterations.....................: 15136   3017.4609/s
    kafka.writer.dial.count........: 151    30.102841/s
    kafka.writer.error.count.......: 0      0/s
    kafka.writer.message.bytes.....: 5.2 MB  1.0 MB/s
    kafka.writer.message.count.....: 30272   6034.9218/s
    kafka.writer.rebalance.count...: 0      0/s
    kafka.writer.write.count.......: 30272   6034.9218/s
    vus............................: 5      min=5       max=50
    vus_max........................: 50     min=50      max=50

Scale the load

You can easily scale the load by increasing the number of vus. For example, the following command uses 500 vus to load test for a minute:

./k6 run --vus 500 --duration 1m test_script.js

If you are new to k6, check out how to configure the load options in the script or run a stress test with k6.

Extend the test

The script above is all about producing messages to your Kafka server. In fact, you can easily modify the code into a test which produces and consumes messages.

Simply add the following code below the for loop code:

let result = consume(consumer, 10);
check(result, {
    "10 messages returned": (msgs) => msgs.length == 10,
});

The code will read 10 messages each time. Simply modify the value to something higher if you wish to consume more messages.

The output is as follows when you run it with the same command:

running (05.0s), 00/50 VUs, 9778 complete and 0 interrupted iterations
default ✓ [======================================] 50 VUs  5s

    ✓ is sent
    ✓ 10 messages returned

    █ teardown

    checks.........................: 100.00% ✓ 19556      ✗ 0
    data_received..................: 0 B    0 B/s
    data_sent......................: 0 B    0 B/s
    iteration_duration.............: avg=25.53ms min=41.4µs med=18ms max=1.41s p(90)=37.73ms p(95)=52.37ms
    iterations.....................: 9778   1946.80798/s
    kafka.reader.dial.count........: 50     9.955042/s
    kafka.reader.error.count.......: 0      0/s
    kafka.reader.fetches.count.....: 101    20.109184/s
    kafka.reader.message.bytes.....: 15 MB   2.9 MB/s
    kafka.reader.message.count.....: 97830   19478.034846/s
    kafka.reader.rebalance.count...: 0      0/s
    kafka.reader.timeouts.count....: 46     9.158638/s
    kafka.writer.dial.count........: 152    30.263327/s
    kafka.writer.error.count.......: 0      0/s
    kafka.writer.message.bytes.....: 3.4 MB  669 kB/s
    kafka.writer.message.count.....: 19556   3893.615961/s
    kafka.writer.rebalance.count...: 0      0/s
    kafka.writer.write.count.......: 19556   3893.615961/s
    vus............................: 50     min=50      max=50
    vus_max........................: 50     min=50      max=50

Kafka metrics in k6

By default, k6 has its own built-in metrics which are collected automatically. Apart from that, you can create your own custom metrics. Custom metrics can be categorized into the following types:

Counter: A metric that cumulatively sums added values.
Gauge: A metric that stores the min, max and last values added to it.
Rate: A metric that tracks the percentage of added values that are non-zero.
Trend: A metric that allows for calculating statistics on the added values (min, max, average and percentiles).

Besides k6, k6 extensions can collect metrics and report them as part of the k6 results output. In this case, xk6-kafka collects individual stats for both reader and writer.

Reader

Let’s have a look at the metrics meant for the reader.

Metrics	Type	Description
kafka.reader.dial.count	Counter	Total number of times the reader tries to connect to Kafka.
kafka.reader.error.count	Counter	Total number of errors occurred when reading from Kafka.
kafka.reader.fetches.count	Counter	Total number of times the reader fetches batches of messages from Kafka.
kafka.reader.message.bytes	Counter	Total bytes consumed.
kafka.reader.message.count	Counter	Total number of messages consumed.
kafka.reader.rebalance.count	Counter	Total number of rebalances of a topic in a consumer group (deprecated) .
kafka.reader.timeouts.count	Counter	Total number of timeouts occurred when reading from Kafka

Writer

As for the writer, the metrics are as follows:

Metrics	Type	Description
kafka.writer.dial.count	Counter	Total number of times the writer tries to connect to Kafka.
kafka.writer.error.count	Counter	Total number of errors occurred when writing to Kafka.
kafka.writer.message.bytes	Counter	Total bytes produced.
kafka.writer.message.count	Counter	Total number of messages produced.
kafka.writer.rebalance.count	Counter	Total number of rebalances of a topic (deprecated).
kafka.writer.write.count	Counter	Total number of times the writer writes batches of messages to Kafka.

There are more available kafka metrics, as you can find them here. However, the extension does not collect all metrics yet. You can follow this GitHub issue to track the progress of their additions.

More examples

Moreover, the xk6-kafka repository provides a few test scripts that work out-of-the-box for new users. At the time of this writing, it comes with the following tests:

test_avro.js : Tests Kafka with 200 Avro messages per iteration.
test_avro_with_schema_registry.js : Tests Kafka with 200 Avro messages per iteration using schema registry
test_json.js : Tests Kafka with 200 JSON messages per iteration.
test_json_with_snappy_compression.js : Tests Kafka with 200 JSON messages per iteration using snappy compression
test_sasl_auth.js : Tests Kafka with 200 JSON messages per iteration and based on SASL authentication.
test_topics.js : List topics on all Kafka partitions and create random topics.

Feel free to experiment with them and modify the code accordingly based on your own use cases. If you bump into issues, report them on GitHub.

Conclusion

In conclusion, load testing Apache Kafka is now a lot easier with k6. k6 provides the foundation to create and scale your load tests, and the xk6-kafka extension brings a convenient API to interact with a Kafka server.

If you wish to find out more on other available k6 extensions, simply head over to the bundle builder page. The page also allows you to generate the corresponding command for building your own custom k6 binary.

If you have any questions or are interested in building an extension, join the k6 community on Slack.

How to Perform Load Testing with k6 using AWS CodeBuild Platform

Michael Wanyoike — Mon, 13 Sep 2021 12:50:07 +0000

You can find a collection of k6 scripts and build specification for CodeBuild in this tutorial here.

📖What you will learn

How to integrate load testing with k6 into AWS CodeBuild Platform.

Different implementation paths, and when to use each.

Introduction

In this tutorial, we will look into how to integrate performance testing in your development process with AWS CodeBuild and k6.

k6 is an open-source load testing tool for testing the performance of APIs, microservices, and websites. Developers use k6 to test a system's performance under a particular load to catch performance regressions or errors.

AWS CodeBuild is a fully managed continuous integration service hat compiles source code, runs tests, and produces software packages that are ready to deploy. It uses a pay-as-you-go pricing model with 100 free build minutes per month on the AWS Free Tier plan.

Documentation for AWS CodeBuild is tightly bound with other AWS services which makes it difficult to understand AWS CodeBuild as a standalone product. To put it simply, AWS has multiple products in it's CI/CD offering which includes:

AWS CodeBuild : continuos integration service
AWS CodeDeploy : managed deployment service
AWS CodePipeline : continuos deployment service
AWS CodeCommit : managed source control service

You can read this blog post to understand how all these services are integrated to form a complete CI/CD pipeline. Fortunately for us, we only need to concern ourselves with AWS CodeBuild and Amazon EventBridge in order to automate k6 load testing. We recommend having a look at the following links:

Writing your performance test

In order to learn how to write a performance test, we'll work in small steps and improving the existing script as our knowledge develops. The test will be made up of the following:

An HTTP request against our system under test.
A load configuration controlling the test duration and amount of virtual users.
A performance goal, or service level objective, expressed as a threshold.

Creating the test script

The test script we will write first will simulate a virtual user accessing a specified website. To ensure we don't overload the system under test, we'll put the virtual user to sleep for one second before continuing.

// ./test.js
import http from 'k6/http';
import { sleep } from 'k6';

export default function () {  
    const res = http.get('https://test.k6.io');  
    sleep(1);
}

Configuring the load

In this step, we'll look at how we can define parameters. For this example, we'll have 50 virtual users accessing the specified website for the duration of one minute. Due to the sleep we specified earlier, the script will generate 50 interactions per second, resulting in a total of about 2900 iterations.

// ./test.js
import http from 'k6/http';
import { sleep } from 'k6';

export const options = {  
    duration: '1m',  
    vus: 50,
};

export default function () {  
    const res = http.get('https://test.k6.io');  
    sleep(1);
}

If you have installed k6 in your local machine, you can run your test locally in your terminal using the command: k6 run test.js.

Configuring our thresholds

Next, we need to define our Service Level Objectives, also known as SLOs. This is basically a measurement that specifies the desired performance levels at a given load. With k6, this is done using Thresholds where you can define SLOs as Pass/Fail criteria.

If any of the thresholds you define fail your test, k6 will halt and return a non-zero exit code, communicating to the CI tool that the step has failed. Let's see how we can add a Threshold to our previous script. We'll ensure that the 95th percentile response time remains below 500ms.

// ./test.js
import http from 'k6/http';
import { sleep } from 'k6';

export const options = {  
    duration: '1m',  
    vus: 50,  
    thresholds: {    
        http_req_duration: ['p(95)<500'],  
    },
};

export default function () {  
    const res = http.get('https://test.k6.io');  
    sleep(1);
}

Thresholds are powerful features that allow us to define different types of Pass/Fail criteria in the same test run. For example:

The 99th percentile response time must be below 700 ms.
The 95th percentile response time must be below 400 ms.
No more than 1% failed requests.
The content of a response must be correct more than 95% of the time.

Take a look at our Thresholds documentation for additional details on the API and its usage.

Setting up the CodeBuild workflow

We'll use Docker to run load testing in AWS CodeBuild's environment:

docker run -i loadimpact/k6 run - <scripts/test.js

In order to automate the build process, we'll need to create a buildspec.yml file with the following configuration:

version: 0.2

phases:
  pre_build:
    commands:
      - echo Pull loadimpact/k6 image..
      - docker pull loadimpact/k6
  build:
    commands:
      - echo Run performance test
      - docker run -i loadimpact/k6 run - <scripts/test.js
  post_build:
    commands:
      - echo Performance test complete

At the time of writing AWS CodeBuild supports repositories hosted on GitHub, BitBucket, Amazon S3 and AWS CodeCommit. For reference, we'll use GitHub for this tutorial. Once you've prepared a project containing the above files, you'll need to commit and push the changes to your remote repository.

Once your remote repository is ready, you can start working on your AWS Console. Using the Services menu, locate AWS CodeBuild and open the page:

Click the Create project button and populate the form as follows:

Project Configuration
- Project name: Load-Test-Example
- Description: Load testing example with k6
Source
- Source provider: Github
- Repository: 'Public' or 'Repository in my GitHub account' for private
- Repository URL: https://github.com/brandiqa/k6-example-aws-codebuild
- Service Role Permissions: Set true for 'Allow AWS CodeBuild to modify this service role so it can be used with this build project'
Environment
- Operating system: Ubuntu
- Runtime: Standard
- Privileged: true. If you don't enable privilege, the docker command won't have the system level permissions to run.
- Leave all other settings in their default state
Buildspec
- Select 'Use a buildspec file'

Leave all other settings in the default state. Ensure CloudWatch logs is enabled. Click the Create build project button.

The next step is to run the project which you can do using the Start build project. Once it's complete, you should expect the following results:

Running k6 cloud tests

There are two common execution modes to run k6 tests as part of the CI process.

Locally on the CI server.
In k6 cloud, from one or multiple geographic locations.

You might want to use cloud tests in these common cases:

If you're going to run a test from multiple geographic locations (load zones).
If you're going to run a high-load test, that will need more compute resources than available in the runner.

If any of those reasons fit your needs, then running k6 cloud tests is the way to go for you.

⚠️ Try it locally first

Before we start with the configuration, it is good to familiarize ourselves with how cloud execution works, and we recommend you to test how to trigger a cloud test from your machine.

Check out the cloud execution guide to learn how to distribute the test load across multiple geographic locations and more information about the cloud execution.

Now, we will show how to trigger cloud tests using AWS CodeBuild. If you do not have an account with k6 Cloud already, you should go here and start your free trial. After that, get your account token from the cloud app and save it somewhere temporarily.

Next, let's create a modified version of the testing script and call it cloud-test.js:

import http from 'k6/http';
import { sleep } from 'k6';


export const options = {
    ext: {
        loadimpact: {
            name: 'cloud-test',
        },
    },
    duration: '1m',
    vus: 50,
    thresholds: {
        http_req_duration: ['p(95)<500'],
    },
};

export default function () {
    const res = http.get('https://test.k6.io');
    sleep(1);
}

The ext property enables passing of the script filename which is required for cloud execution. The next step is to update buildspec.yml as follows:

version: 0.2

env:
  exported-variables:
    - K6_CLOUD_TOKEN

phases:
  pre_build:
    commands:
      - echo Pull loadimpact/k6 image..
      - docker pull loadimpact/k6
  build:
    commands:
      - echo Run performance test
      - docker run -i -e K6_CLOUD_TOKEN=${K6_CLOUD_TOKEN} loadimpact/k6 cloud - <scripts/cloud-test.js
  post_build:
    commands:
      - echo Performance test complete

You'll need to commit and push the new changes to your remote repository. Next, we need to modify the current build and specify the K6_CLOUD_TOKEN as an environment variable:

After making these changes, run the build. In about a minute or so, you should expect the following output:

And if we copy the highlighted URL and navigate to it in a new tab

Scheduling Builds

It is highly recommended to perform load testing at a time when most users aren't accessing your system. This is because performance testing often places significant stresses on your infrastructure which may cause user experience issues.

To configure scheduled nightly build that runs at a given time of a given day or night, head over to Amazon EventBridge.

Head over to the Amazon EventBridge console and click the Create rule button. Fill in the form as follows:

Name: Schedule-Load-Test-Example
Description: Nightly build
Define pattern: Schedule
Cron Expression: 0 12 * * ? *
Target: CodeBuild project
Project ARN: arn:aws:codebuild:region-ID:account-ID:project/project-name

For the Project ARN, simply replace the values in italics with values from your account i.e.

region-ID: found at the top navigation bar, click the drop-down location menu to see your region id in the format 'us-east-1' or 'us-west-1'
account-ID: follow this guide
project-name: Load-Test-Example

Your fully constructed project ARN should look similar to this:

arn:aws:codebuild:us-east-2:123456789123:project/Load-Test-Example

AWS doesn't fully recognize the standard CRON format. Hence you'll need to use the format specified in their cron guide.

After filling the form, click the 'Create' button to save the changes. The Load-Test-Example CodeBuild project will run at the specified time.

Summary

Building your CI/CD pipeline on AWS platform can be overwhelming at first. Hopefully, this guide should help alleviate the pain by showing you exactly what you need you to configure in order to setup an automated load testing pipeline on AWS.

As demonstrated in this article, you can execute an AWS build that runs k6 either locally or on the cloud. Automating performance testing in your CI/CD pipeline will help you identify issues early before they affect your customers.

How to Perform Load Testing with k6 using Google Cloud Build

Michael Wanyoike — Wed, 01 Sep 2021 10:50:01 +0000

You can find a collection of k6 scripts and Google Cloud Build workflows referenced in this tutorial here.

📖 What you will learn

How to integrate load testing with k6 into Google Cloud Build CI/CD Platform.

Different implementation paths, and when to use each.

Introduction

In this tutorial, we will look into how to integrate performance testing in your development process with Google Cloud Build and k6.

Google Cloud Build is a serverless CI/CD platform that allows developers to create custom workflows for their software development lifecycle within the Google Cloud Platform(GCP) ecosystem. It uses a pay-as-you-go pricing model with 120 free minutes given everyday.

If you've not used Google Cloud Build before, we recommend looking at the following links to get a hold of how it works:

Writing your performance test

We'll start small by writing a simple test that measure the performance of a single endpoint. As with most, if not all, development efforts, performance testing yields the best results if we work in small increments, iterating and expanding as our knowledge increases.

Our test will consist of three parts:

An HTTP request against our system under test.
A load configuration controlling the test duration and amount of virtual users.
A performance goal, or service level objective, expressed as a threshold.

Creating the test script

When we execute our test script, each virtual user will execute the default function as many times as possible until the duration is up. To make sure we don't flood our system under test, we'll make the virtual user sleep for a second before it continues.

// ./test.js
import http from 'k6/http';
import { sleep } from 'k6';

export default function () {
    const res = http.get('https://test.k6.io');
    sleep(1);
}

Configuring the load

We'll configure our test to run 50 virtual users continuously for one minute. Because of the sleep we added earlier, this will result in just below 50 iterations per second, giving us a total of about 2900 iterations.

// ./test.js
import http from 'k6/http';
import { sleep } from 'k6';

export const options = {  
    duration: '1m',  
    vus: 50,
};

export default function () {  
    const res = http.get('https://test.k6.io');  
    sleep(1);
}

If you have installed k6 in your local machine, you can run your test locally in your terminal using the command: k6 run test.js.

Configuring our thresholds

The next step is to define your service level objectives, or SLOs around your application performance. SLOs are a vital aspect of ensuring the reliability of your systems and applications. If you do not currently have any defined SLAs or SLOs, now is an excellent time to consider your requirements.

You can define SLOs as Pass/Fail criteria with Thresholds in your k6 script. k6 evaluates them during the test execution and informs about the Threshold results. If any of the thresholds in our test fails, k6 will return with a non-zero exit code, communicating to the CI tool that the step has failed.

Now, we will add one Threshold to our previous script to validate than the 95th percentile response time must be below 500ms.

// ./test.js
import http from 'k6/http';
import { sleep } from 'k6';

export const options = {  
    duration: '1m',  
    vus: 50,  
    thresholds: {    
        http_req_duration: ['p(95)<500'],  
    },
};

export default function () {  
    const res = http.get('https://test.k6.io');  
    sleep(1);
}

Thresholds are a powerful feature providing a flexible API to define various types of Pass/Fail criteria in the same test run. For example:

The 99th percentile response time must be below 700 ms.
The 95th percentile response time must be below 400 ms.
No more than 1% failed requests.
The content of a response must be correct more than 95% of the time.

Check out the Thresholds documentation for additional details on the API and its usage.

Setting up the Cloud Build workflow

Ideally, the easiest way to setup load testing in a CI/CD environment is to use docker:

docker run -i loadimpact/k6 run - <scripts/test.js

Unfortunately, the above command fails to run within the Cloud Build environment. The solution is to mount the source code files and execute them within a docker container environment. To ensure a smooth CI/CD workflow in Google Build's environment, we'll run our load performance test using docker-compose. First, we need to define our docker-compose.yml file:

version: '3.4'
services:
  k6:
    image: loadimpact/k6:latest
    command: run /scripts/test.js
    volumes:
      - ./scripts:/scripts

Basically, we have defined a single service called k6 that performs the following:

Pulls the k6 docker image from Docker Hub registry
Executes a command that runs the local k6 test
Mounts the scripts folder in the /script path

Next we need to define the CI/CD steps that will be executed on the Cloud Build platform. We'll do this using cloudbuild.yaml file:

steps:
# Load Test
- name: 'gcr.io/cloud-builders/docker'
  args: ['run', '-i', 'loadimpact/k6', 'run', '-', '<scripts/test.js']

Normally, we would have to define steps for install, test, build and deploy. Since we are testing with an app that has already been deployed, test.k6.io, we'll only define a single step, Load Test.

At the time of writing, only GitHub, BitBucket and Google Cloud Source repositories are supported by Google Cloud Platform. For reference, we'll use GitHub for this tutorial. Once you've prepared a project containing the above files, you'll need to commit and push the changes to your remote repository.

Once your remote repository is ready, you can start working on your Google Cloud Console. If you have never used Cloud Build before, you'll need to enable the API first. The setting below is located under API & Services > Library > Developer tools:

By enabling the API, the appropriate IAM service account will be created automatically that will allow you to run the build steps successfully. With Cloud Build, Google provides us with a selection of cloud builders a.k.a execution environments for running our CI/CD steps. These builders include yarn, npm, docker, mvn, go and a few more. Unfortunately, majority of the building tools we need for our projects are not available out of the box.

To add the tools we need, we have to visit the community contributed builders repository where all the tools needed in a software development lifecycle can be found. In our case, we simply need the docker-compose builder. The steps to install this builder in our own Google Cloud account is as follows:

Install and configure gcloud CLI tool to connect with your GCP account from your machine
Run the update command: gcloud components update
Install docker-compose builder on your GCP account using the following commands:

   # Clone the entire cloud builders community repo onto your hard drive
   git clone https://github.com/GoogleCloudPlatform/cloud-builders-community

   # Navigate into the docker-compose folder
   cd cloud-builders-community/docker-compose/

   # Install docker-compose builder into your account
   gcloud builds submit --config=cloudbuild.yaml .

Confirm the docker-compose builder has been installed. See Cloud Build > Container Registry > Images

Once you have completed this step, you can now proceed with creating a Trigger which is the basis of building a CI/CD pipeline in Google Cloud Build.

Head over to Cloud Build > Triggers page and click the Create Trigger button. Populate the form as follows:

Name - Run-Performance-Test
Description - Run k6 load test
Event - Push to a branch
Source - link to your GitHub repository
Branch - main
Configuration - Cloud Build configuration file (yaml or json)
Location - Repository

After hitting save, click the Run button. You should expect the following results:

Running k6 Cloud tests

There are two common execution modes to run k6 tests as part of the CI process.

Locally on the CI server.
In k6 Cloud, from one or multiple geographic locations.

You might want to use cloud tests in these common cases:

If you're going to run a test from multiple geographic locations (load zones).
If you're going to run a high-load test, that will need more compute resources than available in the runner.

If any of those reasons fit your needs, then running k6 Cloud tests is the way to go for you.

⚠️ Try it locally first

Before we start with the configuration, it is good to familiarize ourselves with how cloud execution works, and we recommend you to test how to trigger a cloud test from your machine.

Check out the cloud execution guide to learn how to distribute the test load across multiple geographic locations and more information about the cloud execution.

Now, we will show how to run cloud tests on the Google Cloud Build platform. If you do not have an account with k6 Cloud already, you should go here and start your free trial. After that, get your account token from the cloud app and save it somewhere temporarily.

Next, we need to update our docker-compose.yml file as follows:

version: '3.4'
services:
  k6:
    image: loadimpact/k6:latest
    command: cloud /scripts/test.js
    volumes:
      - ./scripts:/scripts
    environment:
      - K6_CLOUD_TOKEN=${K6_CLOUD_TOKEN}

There are only 2 changes we have made here:

In the command section, we've changed from run to cloud
We've added an environment section. The ${K6_CLOUD_TOKEN} environment variable value needs to be defined outside the docker container, in this case, we'll do it in the cloudbuild.yaml config file where execution call is made.

With most platforms, there is a system for securely storing environment variables. Unfortunately for Google Cloud Build, there isn't. Instead, what we'll paste the token value in plain format within the Trigger configuration. We'll use the inline YAML editor. Simply edit the current Trigger and update the following settings:

Configuration Location: Inline
Click Open Editor button and copy the following:

  steps:
    - name: gcr.io/$PROJECT_ID/docker-compose
      env:
        - 'K6_CLOUD_TOKEN=<paste your token here>'
      args:
        - run
        - k6

Paste in your token key as indicated. Click Done and then Save.

Once we commit and push the changes we made earlier, k6 will run our test using the k6 Cloud, and output the URL to our test results as part of the workflow logs:

And if we copy the highlighted URL and navigate to it in a new tab

Nightly Builds

Triggering a subset of performance tests at a specific time is a best-practice for automating your performance testing.

It's common to run some performance tests during the night when users do not access the system under test. For example, to isolate more extensive tests from other types of testing or to generate a performance report periodically.

To configure scheduled nightly build that runs at a given time of a given day or night, head over to your Trigger configuration and edit as follows:

Event: Manual invocation

Go back to the main Triggers page and click on the menu (vertical ellipses) located at the right end of the row containing your trigger. You should see a new option called Run on schedule. Click on it to start setting up the schedule. You'll need to:

Enable the Schedule API
Select a service account (accept the default which should start with cloud-build-trigger-scheduler@..)
Provide name, description and frequency of the schedule in POSIX cron syntax. Here is an interactive tool for creating crontab scheduling expressions. You can also select your desired time zone.

You can use the following values for reference:

Once you save your schedule settings, Google Cloud Build will take care of running the workflow at the time intervals you specified.

You can also visit the Cloud Scheduler page and trigger the scheduled job manually. In case this fails, edit the Job Schedule and check the Configure the job's target > Body field. By default the branch name is set to master. If you are using a different branch such as main, you need to update as follows:

{"branchName":"main"}

After saving, you should be able to successfully run the job manually.

Summary

While there is a learning curve with setting up a full CI/CD pipeline for your project with Google Cloud, integrating k6 performance tests into a new or existing Google Cloud Build pipeline is quick and easy as only a couple of configuration files are involved.

The configurations mentioned throughout the article, provide the same flexibility and capabilities as you're used to from running k6 locally. By integrating performance tests into your CI/CD pipelines, you'll be able to identify and correct performance regressions as they occur.

Benchmarking Redis with k6

Ng Wai Foong — Wed, 18 Aug 2021 01:47:15 +0000

Previously, I have covered an article on Load Testing SQL Databases with k6. For your information, from k6 version 0.29.0 onwards, you can write a k6 Go extension and build your own k6 binaries. This comes in handy as you can use a single framework for load testing different protocols, such as ZMTQ, SQL, Avro, MLLP, etc.

In this series of k6 extensions, let’s benchmark Redis now. According to redis.io, Redis is a type of in-memory data structure store that can be used as database, cache and message broker.

You might want to evaluate the performance or scalability of Redis instances in given hardware, giving you better insights into the throughput the Redis service can handle.

This tutorial covers Redis performance testing via two different approaches on a Linux machine:

redis-benchmark
xk6-redis

redis-benchmark

By default, Redis comes with its own benchmark utility called redis-benchmark. It is similar to Apache's ab utility and can simulate a number of clients sending a total number of queries simultaneously.

Options

Make sure that you have Redis installed in your system. If you have not done so, kindly head over to the official Redis download page and install it based on the instructions given.

Once you are done with it, you should be able to run the following command:

redis-benchmark --help

You should see the following output:

Usage: redis-benchmark [-h <host>] [-p <port>] [-c <clients>] [-n <requests>] [-k <boolean>]

 -h <hostname>      Server hostname (default 127.0.0.1)
 -p <port>          Server port (default 6379)
 -s <socket>        Server socket (overrides host and port)
 -a <password>      Password for Redis Auth
 -c <clients>       Number of parallel connections (default 50)
 -n <requests>      Total number of requests (default 100000)
 -d <size>          Data size of SET/GET value in bytes (default 3)
 --dbnum <db>       SELECT the specified db number (default 0)
 -k <boolean>       1=keep alive 0=reconnect (default 1)
 -r <keyspacelen>   Use random keys for SET/GET/INCR, random values for SADD
  Using this option the benchmark will expand the string __rand_int__
  inside an argument with a 12 digits number in the specified range
  from 0 to keyspacelen-1. The substitution changes every time a command
  is executed. Default tests use this to hit random keys in the
  specified range.
 -P <numreq>        Pipeline <numreq> requests. Default 1 (no pipeline).
 -e                 If server replies with errors, show them on stdout.
                    (no more than 1 error per second is displayed)
 -q                 Quiet. Just show query/sec values
 --csv              Output in CSV format
 -l                 Loop. Run the tests forever
 -t <tests>         Only run the comma separated list of tests. The test
                    names are the same as the ones produced as output.
 -I                 Idle mode. Just open N idle connections and wait.

Examples

Depending on your needs, a typical example is to just run the benchmark with the default configuration:

redis-benchmark

It is a good idea to use the -q option. Here is an example for running 100k of requests in quiet mode:

redis-benchmark -q -n 100000

In addition, you can run parallel clients via the -c option. The following example use 20 parallel clients for a total of 100k requests:

redis-benchmark -q -n 100000 -c 20

You can restrict the test to run only a subset of the commands. For example, you can use the following command to test only set and get commands:

redis-benchmark -q -t set,get -n 100000

In fact, you can run test on specific commands for benchmarking like the following example:

redis-benchmark -q -n 100000 script load "redis.call('set','key','value')"

If your Redis server is running on a different hostname and port, you can benchmark the server as follows:

redis-benchmark -h 192.168.1.1 -p 6379 -n 100000 -c 20

You should get the following output indicating the requests per second for each of the test conducted:

PING_INLINE: 43478.26 requests per second
PING_BULK: 41666.67 requests per second
SET: 43478.26 requests per second
GET: 43478.26 requests per second
INCR: 40000.00 requests per second
LPUSH: 43478.26 requests per second
RPUSH: 37037.04 requests per second
LPOP: 45454.55 requests per second
RPOP: 34482.76 requests per second
SADD: 43478.26 requests per second
HSET: 45454.55 requests per second
SPOP: 45454.55 requests per second
LPUSH (needed to benchmark LRANGE): 40000.00 requests per second
LRANGE_100 (first 100 elements): 45454.55 requests per second
LRANGE_300 (first 300 elements): 43478.26 requests per second
LRANGE_500 (first 450 elements): 47619.05 requests per second
LRANGE_600 (first 600 elements): 38461.54 requests per second
MSET (10 keys): 41666.67 requests per second

Latency

Sometimes, you might prefer to analyze the latency instead. There are two types of latency measurement provided by redis-cli:

latency
intrinsic latency

In this case, we measure latency as the time between sending a request to Redis and receiving a response. On the other hand, intrinsic latency refers to the system latency that is highly dependent on external factors such as operating system kernel or virtualization. Since Redis 2.8.7, you can measure the intrinsic latency independently.

Please note that you can only run redis-cli in the machine which hosts the Redis server unlike redis-benchmark which is runnable on the client machine. Besides that, this mode is not connected to a Redis server at all and the measurement is based on the largest time in which the kernel does not provide CPU time to run to the redis-cli process itself. As a result, it is not an actual measurement of the latency between client and Redis server.

Having said that, it does provide a quick analysis if there is something wrong with the machine that hosts the Redis server.

Run the following command to get the overall latency of your Redis server:

redis-cli --latency

You should see an increase in the sample as time goes by and the average latency:

min: 0, max: 5, avg: 0.22 (2406 samples)

Use Ctrl+C to stop it as the process will run indefinitely without stopping.

For intrinsic latency, you should use the following command instead:

redis-cli --intrinsic-latency 10

You can pass an integer representing the duration of the test. In this case, the test will run for 10 seconds. The output is as follows:

Max latency so far: 1 microseconds.
Max latency so far: 15 microseconds.
Max latency so far: 16 microseconds.
Max latency so far: 17 microseconds.
Max latency so far: 18 microseconds.
Max latency so far: 20 microseconds.
Max latency so far: 21 microseconds.
Max latency so far: 24 microseconds.
Max latency so far: 25 microseconds.
Max latency so far: 50 microseconds.
Max latency so far: 74 microseconds.
Max latency so far: 87 microseconds.
Max latency so far: 150 microseconds.
Max latency so far: 1089 microseconds.
Max latency so far: 1715 microseconds.
Max latency so far: 2344 microseconds.
Max latency so far: 7438 microseconds.
Max latency so far: 8002 microseconds.

158645097 total runs (avg latency: 0.0630 microseconds / 63.03 nanoseconds per run).
Worst run took 126948x longer than the average latency.

The average latency is about 0.22 milliseconds while the intrinsic latency is 0.063 microseconds.

Let’s proceed to the next section and start exploring another testing approach using k6.

xk6-redis

k6 provides the capabilities to do performance testing with scripting language. This is a big plus to developers and Q&A testers as you will have a better control of the entire workflow of the test. For example, you can ramp up or ramp down the requests at specific intervals of the test which is not achievable when using redis-benchmark.

Fortunately, k6 provides the xk6-redis extension as part of their ecosystem. You can use it directly to build your own custom k6 binaries for testing Redis server.

This extension comes with the following API:

Output	Usage
Client(options)	Represent the Client construtor. Returns a new Redis client object.
client.set(key, value, expiration time)	Set the given key with the given value and expiration time.
client.get(key)	Get returns the value for the given key.

Building k6 with the redis extension

Before that, make sure you have the following installed in your machine:

Once you have completed the installation, run the following to install xk6 module:

go install github.com/k6io/xk6/cmd/xk6@latest

If you have installed xk6 directory to Go module, you can make your Redis k6 build by running:

xk6 build --with github.com/k6io/xk6-redis

You should get a k6 executable in your current working directory.

Alternatively, you can download the pre-compiled binaries at the following Github repository. The latest version at the time of this writing is v0.4.1. If you have trouble identifying the architecture of your Linux machine, simply run the following command:

dpkg --print-architecture

Let’s say that the command returns the following:

amd64

You should download the xk6_0.4.1_linux_amd64.tar.gz asset and extract it as follows:

tar -xvf xk6_0.4.1_linux_amd64.tar.gz

You should get the following files in your working directory:

README.md
LICENSE
xk6

Then, run the following command to build k6 for Redis:

./xk6 build --with github.com/k6io/xk6-redis

You should have now a new k6 binary in your working directory.

k6 Script

Next, let’s create a new JavaScript file called test_script.js in the same directory as your k6 executable. Append the following import statement at the top of the file:

import redis from 'k6/x/redis';

Continue by adding the following code which connect to your Redis server:

const client = new redis.Client({
  addr: 'localhost:6379',
  password: '',
  db: 0,
});

It accepts the following an object with the following fields:

addr: hostname and port of your Redis server denoted as hostname:port.
password: password of your Redis server.
db: the db number ranging from 0 to 15.

To keep it simple and short, the test case is going to be as follows:

Set a new key:value on start of test.
Running parallel VUs to get the same key repeatedly.

The k6 setup function runs only once at the test start, independently of the test load and duration. Let’s set the key: value as follows:

export function setup() {
  client.set('key', 'value', 0);
}

The set function accepts three input parameters:

key
value
expiration time

Then, define the default function which will be called repeatedly by each VU during the entire test:

export default function () {
  client.get('key');
}

The complete code is as follows:

import redis from 'k6/x/redis';
import { check } from 'k6';

const client = new redis.Client({
  addr: 'localhost:6379',
  password: '',
  db: 0,
});

export function setup() {
  client.set('key', 'value', 0);
}

export default function () {
  client.get('key');
}

Running the test

Save the test script and run the following command to the test your Redis server for 5 seconds:

./k6 run test_script.js --duration 5s

By default, it is using one Virtual User (VU) but you can modify it with the --vus flag. You should see the following output:

          /\      |‾‾| /‾‾/   /‾‾/   
     /\  /  \     |  |/  /   /  /    
    /  \/    \    |     (   /   ‾‾\  
   /          \   |  |\  \ |  (‾)  | 
  / __________ \  |__| \__\ \_____/ .io

  execution: local
    script: test_script.js
    output: -

  scenarios: (100.00%) 1 scenario, 1 max VUs, 35s max duration (incl. graceful stop):
        * default: 1 looping VUs for 5s (gracefulStop: 30s)


running (05.0s), 0/1 VUs, 42037 complete and 0 interrupted iterations
default ✓ [======================================] 1 VUs  5s

    █ setup

    data_received........: 0 B   0 B/s
    data_sent............: 0 B   0 B/s
    iteration_duration...: avg=104.45µs min=53.7µs med=88.6µs max=9.32ms p(90)=115.4µs p(95)=129.5µs
    iterations...........: 42037 8401.691798/s
    vus..................: 1    min=1       max=1
    vus_max..............: 1    min=1       max=1

This test reports that the Redis server handles 8401 iterations per second. Because each iteration refers to one execution of the default function and there is one request call in our default function, the server is handling 8401 GET requests per second in this test.

Scale the load

Let’s increase the load gradually until it encounters an error. For a start, set the VUs to 100 as follows:

./k6 run test_script.js --duration 5s --vus 100

The output is as follows:

running (05.0s), 000/100 VUs, 111939 complete and 0 interrupted iterations
default ↓ [======================================] 100 VUs  5s

    █ setup

    data_received........: 0 B  0 B/s
    data_sent............: 0 B  0 B/s
    iteration_duration...: avg=4.39ms min=46.8µs med=3.32ms max=87.24ms p(90)=9.5ms p(95)=12.51ms
    iterations...........: 111939 22304.954101/s
    vus..................: 100  min=100     max=100
    vus_max..............: 100  min=100     max=100

It indicates that your Redis server can sustain about 22304 iterations per second for 100 users at the same time.

Continue the test and set the VUs to 1000 this time:

./k6 run test_script.js --duration 5s --vus 1000

Depending on the configuration of your Redis, you might encounter the following error:

ERRO[0003] ERR max number of clients reached
running at go.k6.io/k6/js/common.Bind.func1 (native)
default at file:///home/wfng/test_script.js:14:14(4)  executor=constant-vus scenario=default source=stacktrace

It indicates that you have reached the max number of clients allowed. You can check the number of active connection by running the following command inside redis-cli:

info clients

It will returns the following output:

# Clients
connected_clients:7
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

To get the max limit, use the following instead:

config get maxclients

The output is as follows:

1) "maxclients"
2) "500"

Latency

Now, let’s have a look at how to get the latency via k6. At the time of this writing, the xk6-redis extension does not report latency as part of its metrics. However, you can easily extend the code in your script and implement your own custom metrics.

Have a look at the following workaround to measure latency. First, let’s add the following import statement at the top of your k6 script:

import { Trend } from 'k6/metrics';

Then, initialize a Trend instance as follows:

let RedisLatencyMetric = new Trend('redis_latency', true);

It accepts two input arguments:

name: the name of the custom metric.
isTime: a boolean indicating whether the values added to the metric are time values or just untyped values.

Add the final touch by modifying the default function as follows:

export default function () {
  const start = Date.now();
  client.get('key');
  const latency = Date.now() - start;
  RedisLatencyMetric.add(latency);
}

Have a look at the following complete code which initialize the options directly inside the script:

import { Trend } from 'k6/metrics';
import redis from 'k6/x/redis';

let RedisLatencyMetric = new Trend('redis_latency', true);

export let options = {
  vus: 40,
  duration: '10s',
}

const client = new redis.Client({
  addr: 'localhost:6379',
  password: '',
  db: 0,
});

export function setup() {
  client.set('key', 'value', 0);
}

export default function () {
  const start = Date.now();
  client.get('key');
  const latency = Date.now() - start;
  RedisLatencyMetric.add(latency);
}

You should be able to see redis_latency metrics once the test has completed.

iteration_duration...: avg=782.57µs min=67.35µs med=732.92µs max=15.86ms p(90)=1.1ms p(95)=1.3ms
iterations...........: 506755 50660.636169/s
redis_latency........: avg=764.8µs  min=0s     med=1ms     max=16ms    p(90)=1ms   p(95)=1ms

⚠️ Please note that this workaround of measuring the latency is only indicative, as the JavaScript implementation adds an overhead that might skew the reported latency, especially when the latency is in the sub-microsecond range.

It would be great if the xk6-redis extension provided its own built-in Redis latency metrics similar to the HTTP request metrics. Measuring Redis latency in Go directly would be much more accurate and avoid the unnecessary RedisLatencyMetric script code.

Conclusion

All in all, redis-benchmark is a good tool that provides you with a quick glimpse of the performance of your Redis server. On the other hand, k6 is scriptable in JavaScript and can provide you with better control over the execution and workflow of your test. A scripting language is more flexible for testing various ways to connect and query your Redis server.

In fact, you can utilize both of the tools to get the best out of them. For example, you can run redis-benchmark when you install it on your machine for the first time, to get a rough idea of the performance. Subsequently, use k6 for more advanced cases like integrating your test with your existing toolbox or automating your testing.

Load Testing SQL Databases with k6

Ng Wai Foong — Fri, 16 Jul 2021 03:46:34 +0000

This short tutorial shows how to run a k6 test for load testing a database.

In performance testing, we often trigger load tests that simulate realistic user flows, particularly those that are most commonly seen in production. This type of acceptance testing usually interacts with various parts of our infrastructure: web servers, microservices, databases, etc.

But what if you want to test the performance or scalability of an infrastructure resource in isolation?

In many cases, internal components use custom protocols, and the testing tool needs to support those protocols to test the resource individually. Luckily, with k6, you can use or create extensions that allow you to test different protocols, such as ZMTQ, SQL, Avro, MLLP, etc.

One of the components that you might want to test separately is the database. Databases play an essential role in the performance of our applications, and they can be the bottleneck when experiencing a high volume of users.

Load testing the database directly could provide you with better insights about the database performance in advance. As a result, you could thoroughly plan out your database architecture and determine how to scale it properly.

In this tutorial, let’s explore how to load test a database using the xk6-sql extension. For simplicity, the tests will be executed against a local SQLite3 server but the extension supports the following RDBMS databases:

PostgreSQL
MySQL
SQLite3
MS SQL

Build

In this section, you will install all the required components and build a k6 binary for SQL.

Install the C Compiler for SQLite3

A C compiler is required if you are using SQLite3. Simply install the build-essential package if you are using Debian-based operating system. For Windows users, download the tdm-gcc compiler, extract it and place it in any directory that you prefer. Then, add the path of the bin folder to Environment Variable as follows:

Install the Golang Toolchain

Head over to Golang’s installation page and download the installer based on the operating system of your machine. Once you have installed, run the following command to verify the version.



go version

You should get information related the version number of Go as well as your system architecture:



go version go1.16.4 windows/amd64

Build the k6 binary including the SQL extension

For non-SQLite database, run the following command to build the k6 binary:



xk6 build master --with github.com/imiric/xk6-sql

You need to set CGO_ENABLED to 1 when building for SQLite3 to ensure that C compiler is used:



CGO_ENABLED=1 xk6 build master --with github.com/imiric/xk6-sql

On Windows platform, you need set it explicitly, using set first to call the build command:



set CGO_ENABLED=1
xk6 build master --with github.com/imiric/xk6-sql

You should see the following output on your console:



2021/06/17 14:29:43 [INFO] Temporary folder: C:\Users\wfng\AppData\Local\Temp\buildenv_2021-06-17-1429.359000039
2021/06/17 14:29:43 [INFO] Writing main module: C:\Users\wfng\AppData\Local\Temp\buildenv_2021-06-17-1429.359000039\main.go
2021/06/17 14:29:43 [INFO] Initializing Go module
2021/06/17 14:29:43 [INFO] exec (timeout=10s): C:\Program Files\Go\bin\go.exe mod init k6
go: creating new go.mod: module k6
go: to add module requirements and sums:
        go mod tidy
2021/06/17 14:29:44 [INFO] Pinning versions
2021/06/17 14:29:44 [INFO] exec (timeout=0s): C:\Program Files\Go\bin\go.exe get -d -v go.k6.io/k6@master
go: downloading go.k6.io/k6 v0.32.1-0.20210616133500-9f3dd60fbdc1
go get: added go.k6.io/k6 v0.32.1-0.20210616133500-9f3dd60fbdc1
2021/06/17 14:30:50 [INFO] exec (timeout=0s): C:\Program Files\Go\bin\go.exe get -d -v github.com/imiric/xk6-sql
go get: added github.com/imiric/xk6-sql v0.0.0-20210517160107-d222ad8b93eb
2021/06/17 14:30:52 [INFO] Build environment ready
2021/06/17 14:30:52 [INFO] Building k6
2021/06/17 14:30:52 [INFO] exec (timeout=0s): C:\Program Files\Go\bin\go.exe mod tidy
2021/06/17 14:30:56 [INFO] exec (timeout=0s): C:\Program Files\Go\bin\go.exe build -o C:\Users\wfng\Documents\k6_test\k6.exe -ldflags -w -s -trimpath
2021/06/17 14:31:15 [INFO] Build complete: .\k6.exe
2021/06/17 14:31:15 [INFO] Cleaning up temporary folder: C:\Users\wfng\AppData\Local\Temp\buildenv_2021-06-17-1429.359000039

After that, you should have a new k6 binary in your working directory. Since I am building on the Windows platform, I got the k6.exe executable file.

k6 Script

You need to write a JavaScript file in order to perform load testing with k6. Let’s have a look at an example of a simple test script for load testing API via HTTP:



import http from 'k6/http';
import { sleep } from 'k6';

export default function () {
  http.get('https://test.k6.io');
  sleep(1);
}

Each test script requires a default function which will be executed over and over again during testing. The script above makes a GET call to our own k6 test API and sleeps for a second on each execution for a single VU.

For load testing a database, all you need to do is to import the SQL module that you have created earlier and write the corresponding code inside the default function.

Create a new JavaScript file called script.js in the same directory as your k6 binary file.

Import SQL module

You can import your newly created SQL module by adding this line to script.js:



import sql from 'k6/x/sql';

The naming is based on what has been defined in the Go file. In this case, it is defined as k6/x/sql.

Connect to database

You can easily connect to your database by calling the sql.open function:



const db = sql.open("sqlite3", "./test.db");

It accepts two input parameters:

type - the type of database (mysql, postgres, sqlite3, sqlserver)
name - the name of the database

Setup and Teardown the database

Before executing a SQL command, let’s explore a little more about the k6 test life cycle. It typically follows this structure:



// 1. init code (call once per VU)

export function setup() {
  // 2. setup code (call once at the beginning of test)
}

export default function (data) {
  // 3. VU code
}

export function teardown(data) {
  // 4. teardown code (call once at the end of test)
}

You can add in any init code right before the setup, default function and teardown. Init code serves as the initialization and will be called once for each virtual user (VU).

Also, you can specify a setup function which is called once at the beginning of the test where VU is 0. On the other hand, teardown is called once at the end of the test.

As explained earlier, the default function serves as the VU code which will be executed continuously during testing.

Execute SQL command

After connecting to the database, you can use the db object and call the exec to run any SQL command.

For example, as part of the setup process, before the “load” runs, you can create a new table and insert a few rows of data into the table as follows:



export function setup() {
  db.exec(`CREATE TABLE IF NOT EXISTS person (
           id integer PRIMARY KEY AUTOINCREMENT,
           email varchar NOT NULL,
           first_name varchar,
           last_name varchar);`);

  db.exec("INSERT INTO person (email, first_name, last_name) VALUES('johndoe@email.com', 'John', 'Doe');");
  db.exec("INSERT INTO person (email, first_name, last_name) VALUES('marysue@email.com', 'Mary', 'Sue');");
  db.exec("INSERT INTO person (email, first_name, last_name) VALUES('dorydoe@email.com', 'Dory', 'Doe');");
}

And you should not forget to clean up the database at the end of the test with the teardown function. This example deletes the table and closes the database connection:



export function teardown() {
  db.exec("DELETE FROM person;");
  db.exec("DROP TABLE person;");
  db.close();
}

Query data from database

You can easily query the output with the query function. Let’s use it as part of load testing to determine how many iterations you can get when querying the database:



export default function () {
  let results = sql.query(db, "SELECT * FROM person;");
}

As usual, you can run the check statement to determine the output. Let’s do a simple check on the total rows of data in the database:



import { check } from 'k6';

...

export default function () {
  let results = sql.query(db, "SELECT * FROM person;");
  check(results, {
    'is length 3': (r) => r.length === 3,
  });
}

The complete script code is as follows:



import sql from 'k6/x/sql';
import { check } from 'k6';

const db = sql.open("sqlite3", "./test.db");

export function setup() {
  db.exec(`CREATE TABLE IF NOT EXISTS person (
           id integer PRIMARY KEY AUTOINCREMENT,
           email varchar NOT NULL,
           first_name varchar,
           last_name varchar);`);

  db.exec("INSERT INTO person (email, first_name, last_name) VALUES('johndoe@email.com', 'John', 'Doe');");
  db.exec("INSERT INTO person (email, first_name, last_name) VALUES('marysue@email.com', 'Mary', 'Sue');");
  db.exec("INSERT INTO person (email, first_name, last_name) VALUES('dorydoe@email.com', 'Dory', 'Doe');");
}

export function teardown() {
  db.exec("DELETE FROM person;");
  db.exec("DROP TABLE person;");
  db.close();
}

export default function () {
  let results = sql.query(db, "SELECT * FROM person;");
  check(results, {
    'is length 3': (r) => r.length === 3,
  });
}

Running the test

Once we have the completed script, you can run the test. Let’s start running the load test for 5 seconds:



k6 run script.js --duration 5s

By default, it is using just one virtual user (VU) but you can modify it via --vus flag. You should see the following output:



          /\      |‾‾| /‾‾/   /‾‾/
     /\  /  \     |  |/  /   /  /
    /  \/    \    |     (   /   ‾‾\
   /          \   |  |\  \ |  (‾)  |
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: script.js
     output: -

  scenarios: (100.00%) 1 scenario, 1 max VUs, 35s max duration (incl. graceful stop):
           * default: 1 looping VUs for 5s (gracefulStop: 30s)


running (05.1s), 0/1 VUs, 34467 complete and 0 interrupted iterations
default ✓ [======================================] 1 VUs  5s

     ✓ is length 3

     █ setup

     █ teardown

     checks...............: 100.00% ✓ 34467       ✗ 0
     data_received........: 0 B     0 B/s
     data_sent............: 0 B     0 B/s
     iteration_duration...: avg=143.57µs min=0s med=0s max=43.24ms p(90)=519.2µs p(95)=985.47µs
     iterations...........: 34467   6812.032587/s
     vus..................: 1       min=1         max=1
     vus_max..............: 1       min=1         max=1

In this case, it shows that the database can handle about 6812 queries per second and average time of 144µs per iteration.

Scale the load

In the previous test, you have specified just a single virtual user. Let’s scale it up to 10 and see how SQLite performs. Run the following command:



k6 run script.js --duration 5s --vus 10

You should get the following result:



running (05.1s), 00/10 VUs, 43228 complete and 0 interrupted iterations
default ✓ [======================================] 10 VUs  5s

    ✓ is length 3

    █ setup

    █ teardown

    checks...............: 100.00% ✓ 43228    ✗ 0
    data_received........: 0 B  0 B/s
    data_sent............: 0 B  0 B/s
    iteration_duration...: avg=1.16ms min=0s med=0s max=136.03ms p(90)=522.5µs p(95)=570.15µs
    iterations...........: 43228   8446.461494/s
    vus..................: 10   min=10      max=10
    vus_max..............: 10   min=10      max=10

Let’s continue the test and set the VU to 100 this time.



k6 run script.js --duration 5s --vus 100

The output is as follows:



default ✓ [======================================] 100 VUs  5s

    ✓ is length 3

    █ setup

    █ teardown

    checks...............: 100.00% ✓ 97490        ✗ 0
    data_received........: 0 B  0 B/s
    data_sent............: 0 B  0 B/s
    iteration_duration...: avg=5.07ms min=0s med=506.55µs max=140.07ms p(90)=18.13ms p(95)=28.58ms
    iterations...........: 97490   19034.709634/s
    vus..................: 100  min=100     max=100
    vus_max..............: 100  min=100     max=100

It indicates that SQLite is capable of supporting 100 users with an average duration of 5.07ms per transaction.

For actual use cases, you should continue to scale it up to the point where it will congest your database and cause it to breakdown. This allows you to have a better idea on the maximum limit of your database.

If you are new to k6, check out how to configure the load options in the script or run a stress test with k6.

About k6 Extensions

For your information, you can combine multiple extensions and build your own custom k6 binary. For example, you can use the following command to build a k6 binary for both sql and redis:



xk6 build v0.32.0 --with github.com/dgzlopes/xk6-redis --with github.com/imiric/xk6-sql

Simply head over to the bundle builder page to generate the corresponding command based on your own use cases.

With this tutorial, I wanted to show you how easy it is to load test your database or other dependencies separately with k6. If you have any questions or are interested in building an extension, join the k6 community on Slack.

Intro to Testing ASP.NET APIs with k6 - When Unit Tests Meet Load Testing

Carlos Schults — Thu, 06 May 2021 08:41:29 +0000

Software organizations are always on the hunt for ways to satisfy and delight their customers. Application performance is certainly a part of that, and ensuring good performance requires testing. Load testing is one of the types of performance testing and, in this post, you'll see how to use k6 to perform load testing on a .NET RESTful API.

By the end of the post, you’ll have a more solid knowledge of load testing, why it’s important and how it’s performed. You’ll also have practical experience performing load testing with k6, and you’ll understand some of the unique ways in which this tool sets itself apart from competitors. With all of that being said, let’s get started!

Load Testing: The What, Why, and How

Before getting our hands dirty, let’s start with some fundamentals. What is load testing?

Defining Load Testing

Load testing is one of the many types of web performance testing. It consists of verifying how an application behaves under a heavy load—hence the name. With the help of load testing, you can assess the response of your app to increasing demand and then improve it accordingly.

Why Should You Care About Load Testing?

Why is load testing important? Well, as we’ve explained, when it comes to online services, it’s essential to provide a great user experience to your customers, since the competition is only a few clicks away. And performance is a crucial part of the overall user experience of a website.

But why use load testing specifically? In short, you should adopt load testing when you're concerned about the availability and scalability of your website. Even though the frontend is responsible for relevant performance issues, you should focus on the backend to ensure your performance and reliability goals when scaling your site. And load testing is exactly the tool you need to verify how your website reacts when put under load.

How Do You Perform Load Testing

How is load testing actually performed? As you’ll see, that can vary, even because load testing tools can come in different categories. However, the gist of it is that load testing simulates multiple users sending multiple requests to the app, concurrently. You choose the number of users and requests based on the estimated load you expect your app to be able to handle.

Rolling Up Your Sleeves: Let's Load Test An ASP.NET API

After our brief introduction to load testing, it’s time to learn how to do it in practice.

Obtaining The Requirements

The application we’ll be testing throughout this tutorial is a RESTful API written in ASP.Net Core. To be able to run the application, you’ll need the .NET SDK version 5.0 or later.

Presenting the Sample Application

For this tutorial, our sample application will be the sample API developed using ASP.Net Core, demonstrated in this tutorial.

Download or clone the complete application from this GitHub repository.

As soon as you get the code, cd into the project’s directory and run the following command:



dotnet run

Now, open your browser and go to localhost:<port>/api/TodoItems to see the resources that are returned.

Let's Start Testing

Here’s where the fun begins: we’ve gotten ourselves an API, so let’s test it.

One thing to keep in mind when doing performance testing is that, as we’ve mentioned, it comes in different types. More specifically, we can split load testing tools into two main camps: scriptable and non-scriptable ones.

Scriptable tools are those that allow you to use a full-fledged scripting language to write your tests. Such tools might offer a steeper learning curve and, in some situations, consume slightly more resources. However, the upside is that they enable users to write flexible and more complex tests.

On the other end of the spectrum, you have non-scriptable tools. They tend to be simpler and consume fewer resources. But on the other hand, they're way less powerful when it comes to the capabilities they offer.

This post is about k6, which is a powerful scriptable tool. But we won’t jump into k6 right away. Instead, we’ll start by testing our API with a non-scriptable tool. That way, you’ll be able to appreciate the differences between those two categories of tools.

Testing a GET Endpoint With a Non-Scriptable Tool

As I’ve mentioned earlier, it’s possible to divide load testing tools into two broad groups: scriptable and non-scriptable ones. So, to appreciate the qualities of the former group, you need at least a little bit of experience with the latter. That’s why before we get to k6 we’ll take a brief detour to show you how testing an endpoint with a non-scriptable tool looks like.

The non-scriptable tool we’ll use is wrk. Feel free to follow along with this part of the tutorial but, since it’s a detour, I won’t provide detailed instructions for it. With that out of the way, let’s begin.

After installing wrk, I’ll run my sample project again by executing:



dotnet run

Then, I’ll use wrk to test the GET endpoint for the TodoItem resource:



./wrk -t12 -c400 -d30s https://localhost:5001/api/TodoItems/

This is exactly the same "basic usage" example taken from the project's README. It runs a 30 seconds test that uses 12 threads keeping 400 connections open. What do the results look like? See in the following image:

Let's unpack that. After a short text summarizing the executed test (30 seconds, 12 threads, 400 connections) we see statistics regarding the latency and requests per second. For each of those we see, among other info, the average value (1), the standard deviation (2), the maximum value (3).

We also see the number of requests that were performed, in how much time, and the total volume of data that was read (5). Finally, we see the total amount of requests per second and the volume of data transferred.

Testing a Get Endpoint with k6

Now it’s time to repeat the previous test, this time using k6.

Installing

Let’s start by installing the tool. If you use macOS, install k6 is a breeze using homebrew:



brew install k6
```

Users of Debian, Ubuntu or other Debian-based distros should run the following commands:
```bash
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 379CE192D401AB61
echo "deb https://dl.bintray.com/loadimpact/deb stable main" | sudo tee -a /etc/apt/sources.list
sudo apt-get update
sudo apt-get install k6
```

If you’re on Windows, you can [download the.msi installer](https://k6.io/docs/getting-started/installation/#windows-msi-installer). Users of other platforms, take a look at the [k6 installation guide](https://k6.io/docs/getting-started/installation/).

After the installation is complete, regardless of OS, you should be able to run {% raw %}`k6 version` to see the current version number and verify the installation was a success.

#### Writing and Running Your First Test Script

To perform k6 tests, you’ll need to create a test script. Test scripts in k6 are written in JavaScript, which is a perfect choice, considering most web developers will already be familiar with the language—as opposed to something like Python, Ruby, or a custom DSL.

Using your favorite editor, create a file called **script.js**. I’ll use Visual Studio Code:

```
code script.js
```

Then, add the following code to the new file:

```javascript
import http from 'k6/http';

export default function () {
  http.get('http://localhost:5000/Api/TodoItems');
}
```

Of course, make sure that you’ve started the API. Also, change the port number if necessary.

After saving the test script, you can test it by running:

```
k6 run script.js
```

The results should look something like this:
```
          /\      |‾‾| /‾‾/   /‾‾/   
     /\  /  \     |  |/  /   /  /    
    /  \/    \    |     (   /   ‾‾\  
   /          \   |  |\  \ |  (‾)  | 
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: script.js
     output: -

  scenarios: (100.00%) 1 scenario, 1 max VUs, 10m30s max duration (incl. graceful stop):
           * default: 1 iterations for each of 1 VUs (maxDuration: 10m0s, gracefulStop: 30s)


running (00m00.1s), 0/1 VUs, 1 complete and 0 interrupted iterations
default ✓ [ 100% ] 1 VUs  00m00.1s/10m0s  1/1 iters, 1 per VU

     data_received..................: 1.8 kB 33 kB/s
     data_sent......................: 816 B  15 kB/s
     http_req_blocked...............: avg=22ms    min=5ms     med=22ms    max=39ms    p(90)=35.6ms   p(95)=37.3ms 
     http_req_connecting............: avg=499.9µs min=0s      med=499.9µs max=999.8µs p(90)=899.82µs p(95)=949.8µs
     http_req_duration..............: avg=3.99ms  min=999.7µs med=3.99ms  max=6.99ms  p(90)=6.39ms   p(95)=6.69ms 
       { expected_response:true }...: avg=3.99ms  min=999.7µs med=3.99ms  max=6.99ms  p(90)=6.39ms   p(95)=6.69ms 
     http_req_failed................: 0.00%  ✓ 0 ✗ 2
     http_req_receiving.............: avg=0s      min=0s      med=0s      max=0s      p(90)=0s       p(95)=0s     
     http_req_sending...............: avg=0s      min=0s      med=0s      max=0s      p(90)=0s       p(95)=0s     
     http_req_tls_handshaking.......: avg=19ms    min=0s      med=19ms    max=38ms    p(90)=34.2ms   p(95)=36.1ms 
     http_req_waiting...............: avg=3.99ms  min=999.7µs med=3.99ms  max=6.99ms  p(90)=6.39ms   p(95)=6.69ms 
     http_reqs......................: 2      37.030248/s
     iteration_duration.............: avg=54ms    min=54ms    med=54ms    max=54ms    p(90)=54ms     p(95)=54ms   
     iterations.....................: 1      18.515124/s
```
#### Improving The Test

You’ve just written and run your first k6 test script. Congrats! However, we need to beef it up a little bit so it compares with the test we’ve done using wrk. Our k6 test is, at the moment, a simple request. It sends a get request to the specified URL and then waits for 1 second.

When running a test with k6, you can pass parameters to the test. Run the following command:

```
k6 --vus 400 --duration 30s run script.js
```

You can see we’ve used two parameters here.  The **--duration** option defines that the test should run for 30 seconds.

The **--vus** option stands for [virtual users](https://k6.io/docs/misc/glossary/#virtual-users), which are used to send the requests to the system under test.To understand the concept of VUs, think of them like parallel infinite loops. If you go back to our script, you'll see there is a `default` function. Scripts in k6 need to have that `default` function as a requirement. That's the entry point for the test script. The code inside such a function is what the VUs execute; it gets executed over and over again, for the whole duration of the tests.

Besides the `default` function, you can have additional code. However, code outside `default` is run only once per VU. You use that code for initialization duties, such as importing different modules or loading something from your filesystem.

That being said, let’s see what the result of the command’s execution looks like:

```
          /\      |‾‾| /‾‾/   /‾‾/   
     /\  /  \     |  |/  /   /  /    
    /  \/    \    |     (   /   ‾‾\  
   /          \   |  |\  \ |  (‾)  | 
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: script.js
     output: -

  scenarios: (100.00%) 1 scenario, 400 max VUs, 1m0s max duration (incl. graceful stop):
           * default: 400 looping VUs for 30s (gracefulStop: 30s)


running (0m00.9s), 400/400 VUs, 8 complete and 0 interrupted iterations
default   [   3% ] 400 VUs  00.9s/30s

...

running (0m30.0s), 000/400 VUs, 202254 complete and 0 interrupted iterations
default ✓ [ 100% ] 400 VUs  30s

     data_received..................: 60 MB  2.0 MB/s
     data_sent......................: 36 MB  1.2 MB/s
     http_req_blocked...............: avg=1.66ms   min=0s    med=0s      max=2.68s p(90)=0s      p(95)=0s     
     http_req_connecting............: avg=213.12µs min=0s    med=0s      max=1.32s p(90)=0s      p(95)=0s     
     http_req_duration..............: avg=27.65ms  min=0s    med=24ms    max=1.96s p(90)=40ms    p(95)=54.99ms
       { expected_response:true }...: avg=27.65ms  min=0s    med=24ms    max=1.96s p(90)=40ms    p(95)=54.99ms
     http_req_failed................: 0.00%  ✓ 0     ✗ 404508
     http_req_receiving.............: avg=703.91µs min=0s    med=0s      max=1.35s p(90)=0s      p(95)=999.1µs
     http_req_sending...............: avg=167.24µs min=0s    med=0s      max=1.39s p(90)=0s      p(95)=0s     
     http_req_tls_handshaking.......: avg=1.3ms    min=0s    med=0s      max=2.65s p(90)=0s      p(95)=0s     
     http_req_waiting...............: avg=26.77ms  min=0s    med=24ms    max=1.46s p(90)=39.99ms p(95)=51ms   
     http_reqs......................: 404508 13474.167948/s
     iteration_duration.............: avg=59.23ms  min=998µs med=49.99ms max=2.84s p(90)=78.99ms p(95)=98.99ms
     iterations.....................: 202254 6737.083974/s
     vus............................: 380    min=380 max=400 
     vus_max........................: 400    min=400 max=400 
```
What you see above is what the results look like for me after running the latest version of the command. As you can see, there’s a lot going on here. Let’s walk through some of the main pieces of information:

*   `data_received`: that’s the total amount of data received (109 MB) at a rate of 3.4 MB per second;
*   `http_req_duration`: information about the duration of the HTTP requests performed, including the average, median, minimum and maximum values;
*   `http_req_failed`: the rate of requests that failed;
*   `http_req_waiting`: time spent waiting for the server’s response.

To learn more about the metrics displayed in k6’s results, you can [refer to the documentation](https://k6.io/docs/using-k6/metrics/).

### Testing With a Higher Load

We’ll now perform a different test, sending a PUT request instead of a GET one. We’ll also use an additional option of k6 that allows us to simulate an increase in the load on our application.

For that, I’ll create a new test script and name it script2.js. Here’s what its code looks like:

```javascript
import http from 'k6/http';

const url = 'https://localhost:5001/api/TodoItems/';

export let options = {
  stages: [
    { duration: '15s', target: 100 },
    { duration: '30s', target: 500 },
    { duration: '15s', target: 0 },
  ],
};  

export default function () {

  const headers = { 'Accept': '*/*', 'Content-Type': 'application/json', 'Host': 'localhost:5001' };
  const data = {
      name: 'Send e-mail',
      isComplete: false      
  };

  http.put(url, JSON.stringify(data), { headers: headers });
}
```

As you can see, in the script above we’re sending a PUT request to our API, trying to add a new TodoItem resource. Before that, though, we create an `options` variable in which we define three stages for our test. In the first one, which lasts for 15 seconds, k6 will increase the number of VUs from 0 to 100. Then, it will ramp up to 500 for another 30seconds, before ramping down to 0 in the last 15 seconds.

Here’s what the results look like:

```

          /\      |‾‾| /‾‾/   /‾‾/   
     /\  /  \     |  |/  /   /  /    
    /  \/    \    |     (   /   ‾‾\  
   /          \   |  |\  \ |  (‾)  | 
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: script2.js
     output: -

  scenarios: (100.00%) 1 scenario, 500 max VUs, 1m30s max duration (incl. graceful stop):
           * default: Up to 500 looping VUs for 1m0s over 3 stages (gracefulRampDown: 30s, gracefulStop: 30s)


running (0m00.8s), 006/500 VUs, 6743 complete and 0 interrupted iterations
default   [   1% ] 006/500 VUs  0m00.8s/1m00.0s

...

running (1m00.0s), 000/500 VUs, 671850 complete and 0 interrupted iterations
default ✓ [ 100% ] 000/500 VUs  1m0s

     data_received..............: 32 MB   537 kB/s
     data_sent..................: 84 MB   1.4 MB/s
     http_req_blocked...........: avg=73.54µs min=0s med=0s      max=530.01ms p(90)=0s      p(95)=0s     
     http_req_connecting........: avg=16.56µs min=0s med=0s      max=212ms    p(90)=0s      p(95)=0s     
     http_req_duration..........: avg=19.56ms min=0s med=14ms    max=307.99ms p(90)=41.99ms p(95)=55ms   
     http_req_failed............: 100.00% ✓ 671850 ✗ 0    
     http_req_receiving.........: avg=34.6µs  min=0s med=0s      max=229.99ms p(90)=0s      p(95)=0s     
     http_req_sending...........: avg=85.33µs min=0s med=0s      max=157ms    p(90)=0s      p(95)=999µs  
     http_req_tls_handshaking...: avg=56.37µs min=0s med=0s      max=364ms    p(90)=0s      p(95)=0s     
     http_req_waiting...........: avg=19.44ms min=0s med=14ms    max=272.99ms p(90)=41ms    p(95)=54.99ms
     http_reqs..................: 671850  11196.657731/s
     iteration_duration.........: avg=20.07ms min=0s med=14.99ms max=632.99ms p(90)=42.99ms p(95)=56.99ms
     iterations.................: 671850  11196.657731/s
     vus........................: 1       min=1    max=500
     vus_max....................: 500     min=500  max=500
```

## Here's Where a Scriptable Tool Shines: Testing a User Scenario

Up until now, we’ve been testing using fairly simple scripts. You might be wondering what’s all the fuss about scriptable tools after all. Well, here’s the thing: a non-scriptable tool might be enough if you only need to perform basic requests.

However, there are scenarios in which a non-scriptable tool can really make a difference. One of those is where you need to test a user scenario. In other words, you might need to verify a real usage workflow in your app. You simply can’t do that with a non-scriptable tool.

Up until now, we've been testing an endpoint in isolation. Often, when monitoring your services, you might have found a REST API underperforming at a certain load level and want to simulate this behavior again. A non-scriptable tool is often enough for this type of testing.

Such a type of verification, however, isn’t the most realistic. Why? Well, users don’t behave like that. They don’t do things in a completely isolated way. In the real world, your application continuously responds to a flow of real-user interactions, e.g., visit a page, log in, list items, purchase some items, and so on.

Testing real-world scenarios allow you to validate critical business logic or the most frequent user flows. If you want to mimic this type of interaction, a scriptable tool makes this job possible and more manageable.

The following script is a simple example of how testing a user workflow could look like. Using k6, we hit the GET endpoint, retrieving the existing resources. We then get the id of the first object, and use that to send another get  request, obtaining that resource. After that, we set the value of the `completed` attribute and send a PUT request to update the resource on the server. Finally, we use a check to verify whether the response has the expected status code.

```javascript
import http from 'k6/http';
import { check,  sleep } from 'k6';

export let options = {
    vus: 30,
    duration: '40s'
}

export default function () {
  let url = 'https://localhost:5001/api/TodoItems';

  // getting all todo items
  let response = http.get(url);

  // parsing the response body
  let obj = JSON.parse(response.body);

  // retrieving the id from the first resource
  let idFirstItem = obj[0].id;

  // retrieving the resource with the selected id
  response = http.get(`${url}/${idFirstItem}`);
  let item = JSON.parse(response.body);

  // setting the item as complete
  item.complete = true;

  // updating the item
  const headers = { 'Content-Type': 'application/json' };
  let res = http.put(`${url}/${idFirstItem}`, JSON.stringify(item), { headers: headers });

  // checking the response has the 204 (no content) status code
  check(res, {
    'is status 204': (r) => r.status === 204,
  });

  // random think time between 0 and 5 seconds
  sleep(Math.random() * 5);
}
```

In the previous example, the test runs the same user flow continuously until the test duration ends. But with k6, and you can also simulate different user flows with varying load patterns to run simultaneously. This flexibility allows you to test your application more precisely.

For example, you could run 50 virtual users doing some actions and 100 virtual users doing something different for a different period while generating a constant request rate to one endpoint. 

```javascript
export const options = {

  scenarios: {
    scenario1: {
      executor: 'constant-vus',
      duration: "1m",
      vus: 50,
      exec: "userFlowA"
    },

    scenario2: {
      executor: 'ramping-vus',
      stages: [
        { duration: '1m', target: 100 },
        { duration: '30s', target: 0 },
      ],
      exec: "userFlowB"
    },

    scenario3: {
      executor: 'constant-arrival-rate',
      rate: 200,
      timeUnit: '1s',
      duration: '1m',
      preAllocatedVUs: 50,
      exec: "hammerEndpointA"
    }
  }
}
```

To learn more about configuring advanced traffic patterns, check out the [Scenarios API](https://k6.io/docs/using-k6/scenarios/).


## Here's Where k6 Shines: Setting Performance Goals

The ability to set performance goals for your application is an aspect that differentiates k6 from other load testing tools. With k6, you can verify the performance of your app against expected baselines in a way that’s [not that different from assertions in unit testing](https://k6.io/unit-testing-for-performance/).

How does that work in practice? There are two main options that k6 provides: checks and thresholds. 

[Checks ](https://k6.io/docs/using-k6/checks/) allow you to set expectations in your script and verify those expectations automatically. We’ve used one check in our previous example. Even if the checks fail, the execution of the script doesn’t stop. 

[Thresholds](https://k6.io/docs/using-k6/thresholds/), on the other hand, do interrupt the script’s execution. They’re criteria you can use to fail your load test script when the system under test doesn’t meet the expectations.

Here are some examples of thresholds you could use:

- The system generates at the most 1% errors.
- Response time for 99% of requests should be below 400ms.
- Response time for a specific endpoint must always be below 300ms.

Since k6 allows you to define custom metrics, you can always define thresholds on them as well. Here’s an example of what thresholds look like in an actual script:

```javascript
export let options = {
  thresholds: {
    http_req_failed: ['rate<0.01'],   // http errors should be less than 1% 
    http_req_duration: ['p(99)<400'], // 99% of requests should be below 400ms
  },
};
```

## k6: When Unit Tests Meet Load Testing

In this post, you’ve learned how to get started with load testing using k6. We’ve covered some fundamentals on load testing: you’ve learned the definition of this technique, and the differences between scriptable and non-scriptable load testing tools.

k6 belongs to the latter group. It allows developers to author test scenarios in JavaScript—a language they’re likely to already know. Because of that, k6 is a powerful tool, enabling the performing not only of HTTP benchmarking, but also the verification of realistic usage scenarios.

Perhaps the greatest differentiator of k6 is that it bridges the gap between unit testing and load testing. Developers can use the same workflow they’ve been using for years: creating tests, adding verifications with pass/fail criteria, adding those to the CI/CD pipeline, and then be notified when the tests do fail.

Running distributed k6 tests on Kubernetes

Simme — Thu, 11 Feb 2021 00:00:00 +0000

📖What you will learn

What the operator pattern is and when it is useful

Deploying the k6 operator in your kubernetes cluster

Running a distributed k6 test in your own cluster

⚠️ Experimental

The project used in this article is experimental and changes a lot between commits. Use at your own discretion .

Introduction

One of the questions we often get in the forum is how to run distributed k6 tests on your own infrastructure. While we believe that running large load tests is possible even when running on a single node, we do appreciate that this is something some of our users might want to do.

There are at least a couple of reasons why you would want to do this:

You run everything else in Kubernetes and would like k6 to be executed in the same fashion as all your other infrastructure components.
You have access to a couple of high-end nodes and want to pool their resources into a large-scale stress test.
You have access to multiple low-end or highly utilized nodes and need to pool their resources to be able to reach your target VU count or Requests per Second (RPS).

Prerequisites

To be able to follow along in this guide, you’ll need access to a Kubernetes cluster, with enough privileges to apply objects.

You’ll also need:

The Kubernetes Operator pattern

The operator pattern is a way of extending Kubernetes so that you may use custom resources to manage applications running in the cluster. The pattern aims to automate the tasks that a human operator would usually do, like provisioning new application components, changing the configuration, or resolving problems that occur.

This is accomplished using custom resources which, for the scope of this article, could be compared to the traditional service requests that you would file to your system operator to get changes applied to the environment.

The operator will listen for changes to, or creation of, K6 custom resource objects. Once a change is detected, it will react by modifying the cluster state, spinning up k6 test jobs as needed. It will then use the parallelism argument to figure out how to split the workload between the jobs using execution segments.

Using the k6 operator to run a distributed load test in your Kubernetes cluster

We'll now go through the steps required to deploy, run, and clean up after the k6 operator.

Cloning the repository

Before we get started, we need to clone the operator repository from GitHub and navigate to the repository root:



$ git clone https://github.com/k6io/operator && cd operator

Deploying the operator

Deploying the operator is done by running the command below, with kubectl configured to use the context of the cluster that you want to deploy it to.

First, make sure you are using the right context:



$ kubectl config get-contexts

CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* harley harley harley
          jean jean jean
          ripley ripley ripley

Then deploy the operator bundle using make. This will also apply the roles, namespaces, bindings and services needed to run the operator.



$ make deploy

/Users/simme/.go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
cd config/manager && /Users/simme/.go/bin/kustomize edit set image controller=ghcr.io/k6io/operator:latest
/Users/simme/.go/bin/kustomize build config/default | kubectl apply -f -
namespace/k6-operator-system created
customresourcedefinition.apiextensions.k8s.io/k6s.k6.io created
role.rbac.authorization.k8s.io/k6-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/k6-operator-manager-role created
clusterrole.rbac.authorization.k8s.io/k6-operator-proxy-role created
clusterrole.rbac.authorization.k8s.io/k6-operator-metrics-reader created
rolebinding.rbac.authorization.k8s.io/k6-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/k6-operator-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/k6-operator-proxy-rolebinding created
service/k6-operator-controller-manager-metrics-service created
deployment.apps/k6-operator-controller-manager created

Writing our test script

Once that is done, we need to create a config map containing the test script. For the operator to pick up our script, we need to name the file test.js. For this article, we’ll be using the test script below:



import http from 'k6/http';
import { check } from 'k6';

export let options = {
  stages: [
    { target: 200, duration: '30s' },
    { target: 0, duration: '30s' },
  ],
};

export default function () {
  const result = http.get('https://test-api.k6.io/public/crocodiles/');
  check(result, {
    'http response status code is 200': result.status === 200,
  });
}

Before we continue, we'll run the script once locally to make sure it works:



$ k6 run test.js

If you’ve never written a k6 test before, we recommend that you start by reading this getting started article from the documentation, just to get a feel for how it works.

Let’s walk through this script and make sure we understand what is happening: We’ve set up two stages that will run for 30 seconds each. The first one will ramp up to linearly to 200 VUs over 30 seconds. The second one will ramp down to 0 again over 30 seconds.

In this case the operator will tell each test runner to run only a portion of the total VUs. For instance, if the script calls for 40 VUs, and parallelism is set to 4, the test runners would have 10 VUs each.

Each VU will then loop over the default function as many times as possible during the execution. It will execute an HTTP GET request against the URL we’ve configured, and make sure that the responds with HTTP Status 200. In a real test, we'd probably throw in a sleep here to emulate the think time of the user, but as the purpose of this article is to run a distributed test with as much throughput as possible, I've deliberately skipped it.

Deploying our test script

Once the test script is done, we have to deploy it to the kubernetes cluster. We’ll use a ConfigMap to accomplish this. The name of the map can be whatever you like, but for this demo we'll go with crocodile-stress-test.

If you want more than one test script available in your cluster, you just repeat this process for each one, giving the maps different names.



$ kubectl create configmap crocodile-stress-test --from-file /path/to/our/test.js

configmap/crocodile-stress-test created

⚠️ Namespaces

For this to work, the k6 custom resource and the config map needs to be deployed in the same namespace.

Let’s have a look at the result:



$ kubectl describe configmap crocodile-stress-test

Name: crocodile-stress-test
Namespace: default
Labels: <none>
Annotations: <none>

Data
====
test.js:
----
import http from 'k6/http';
import { check } from 'k6';

export let options = {
  stages: [
    { target: 200, duration: '30s' },
    { target: 0, duration: '30s' },
  ],
};

export default function () {
  const result = http.get('https://test-api.k6.io/public/crocodiles/');
  check(result, {
    'http response status code is 200': result.status === 200,
  });
}

Events: <none>

The config map contains the content of our test file, labelled as test.js. The operator will later search through our config map for this key, and use its content as the test script.

Creating our custom resource (CR)

To communicate with the operator, we’ll use a custom resource called K6. Custom resources behave just as native kubernetes objects, while being fully customizable. In this case, the data of the custom resource contains all the information necessary for k6 operator to be able to start a distributed load test:



apiVersion: k6.io/v1alpha1
kind: K6
metadata:
  name: k6-sample
spec:
  parallelism: 4
  script: crocodile-stress-test

For Kubernetes to know what to do with this custom resource, we first need to specify what API Version we want to use to interpret its content, in this case k6.io/v1alpha1. We’ll then set the kind to K6, and give our resource a name.

As the specification for our custom resource, we now have the option to use a couple of different properties:

Parallelism

Configures how many k6 test runner jobs the operator should spawn.

Script

The name of the config map containing our script.js file.

Separate

Whether the operator should allow multiple k6 jobs to run concurrently at the same node. The default value for this property is false, allowing each node to run multiple jobs.

Arguments

Allowing you to pass arguments to each k6 job, just as you would from the CLI. For instance --tag testId=crocodile-stress-test-1,--out cloud, or —no-connection-reuse.

Deploying our Custom Resource

We will now deploy our custom resource using kubectl, and by that, start the test:



$ kubectl apply -f /path/to/our/k6/custom-resource.yml

k6.k6.io/k6-sample created

Once we do this, the k6 operator will pick up the changes and start the execution of the test. This looks somewhat along the lines of what is shown in this diagram:

Let’s make sure everything went as expected:



$ kubectl get k6 

NAME AGE
k6-sample 2s

$ kubectl get jobs

NAME COMPLETIONS DURATION AGE
k6-sample-1 0/1 12s 12s
k6-sample-2 0/1 12s 12s
k6-sample-3 0/1 12s 12s
k6-sample-4 0/1 12s 12s

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
k6-sample-3-s7hdk 1/1 Running 0 20s
k6-sample-4-thnpw 1/1 Running 0 20s
k6-sample-2-f9bbj 1/1 Running 0 20s
k6-sample-1-f7ktl 1/1 Running 0 20s

The pods have now been created and put in a paused state until the operator has made sure they’re all ready to execute the test. Once that’s the case, the operator deploys another job, k6-sample-starter which is responsible for making sure all our runners start execution at the same time.

Let’s wait a couple of seconds and then list our pods again:



$ kubectl get pods

NAME READY STATUS RESTARTS AGE
k6-sample-3-s7hdk 1/1 Running 0 76s
k6-sample-4-thnpw 1/1 Running 0 76s
k6-sample-2-f9bbj 1/1 Running 0 76s
k6-sample-1-f7ktl 1/1 Running 0 76s
k6-sample-starter-scw59 0/1 Completed 0 56s

All right! The starter has completed and our tests are hopefully running. To make sure, we may check the logs of one of the jobs:



$ kubectl logs k6-sample-1-f7ktl

[...]

Run [100%] paused
default [0%]

Run [100%] paused
default [0%]

running (0m00.7s), 02/50 VUs, 0 complete and 0 interrupted iterations
default [1%] 02/50 VUs 0m00.7s/1m00.0s

running (0m01.7s), 03/50 VUs, 13 complete and 0 interrupted iterations
default [3%] 03/50 VUs 0m01.7s/1m00.0s

running (0m02.7s), 05/50 VUs, 41 complete and 0 interrupted iterations
default [4%] 05/50 VUs 0m02.7s/1m00.0s

[...]

And with that, our test is running! 🎉 After a couple of minutes, we’re now able to list the jobs again to verify they’ve all completed:



$ kubectl get jobs

NAME COMPLETIONS DURATION AGE
k6-sample-starter 1/1 8s 6m2s
k6-sample-3 1/1 96s 6m22s
k6-sample-2 1/1 96s 6m22s
k6-sample-1 1/1 97s 6m22s
k6-sample-4 1/1 97s 6m22s

Cleaning up

To clean up after a test run, we delete all resources using the same yaml file we used to deploy it:



$ kubectl delete -f /path/to/our/k6/custom-resource.yml

k6.k6.io "k6-sample" deleted

Which deletes all the resources created by the operator as well, as shown below:



$ kubectl get jobs
No resources found in default namespace.

$ kubectl get pods
No resources found in default namespace.

⚠️ Deleting the operator

If you for some reason would like to delete the operator altogether, just run make delete from the root of the project..

The idea behind the operator however, is that you let it remain in your cluster between test executions, only applying and deleting the actual K6 custom resources used to run the tests.

Things to consider

While the operator makes running distributed load tests a lot easier, it still comes with a couple of drawbacks or gotchas that you need to be aware of and plan for. For instance, the lack of metric aggregation.

We’ll go through in detail how to set up the monitoring and visualisation of these test runs in a future article, but for now, here’s a list of things you might want to consider:

Metrics will not be automatically aggregated

Metrics generated by running distributed k6 tests using the operator won’t be aggregated, which means that each test runner will produce its own results and end-of-test summary.

To be able to aggregate your metrics and analyse them together, you’ll either need to:

1) Set up some kind of monitoring or visualisation software and configure your K6 custom resource to make your jobs output there.

2) Use logstash, fluentd, splunk, or similar to parse and aggregate the logs yourself.

Thresholds are not evaluated across jobs at runtime

As the metrics are not aggregated at runtime, your thresholds won’t be evaluated using aggregation either. Currently, the best way to solve this is by setting up alarms for passed thresholds in your monitoring or visualisation software instead.

Overpopulated nodes might create bottlenecks

You want to make sure your k6 jobs have enough cpu and memory resources to actually perform your test. Using parallelism alone might not be sufficient. If you run into this issue, experiment with using the separate property.

Experimental

As mentioned in the beginning of the article, the operator is experimental, and as such it might change a lot from commit to commit.

Total cost of ownership

The k6 operator significantly simplifies the process of running distributed load tests in your own cluster. However, there still is a maintenance burden associated with self-hosting. If you'd rather skip that, as well as the other drawbacks listed above, and instead get straight to load testing, you might want to have a look at the k6 cloud offering.

Performance testing gRPC services

Simme — Thu, 12 Nov 2020 00:00:00 +0000

🎉 New in v0.29.0

v0.29.0 contained a lot of interesting features. Have a look at the release notes for details!

📖What you will learn

What gRPC is.

How gRPC differs from JSON-based REST.

Creating and executing your first gRPC performance test using k6.

Outline

What is gRPC
API Types
The proto definition
Getting started
Summary
See also

What is gRPC

gRPC is a light-weight open-source RPC framework. It was originally developed by Google, with 1.0 being released in August 2016. Since then, it's gained a lot of attention as well as a wide adoption.

In comparison to JSON, which is transmitted as human-readable text, gRPC is binary, making it both faster to transmit and more compact. In the benchmarks we've seen between gRPC and JSON-based REST, gRPC has proved to be a lot faster than its more traditional counterpart.

A benchmark by Auth0 reported up to 6 times higher performance, while other benchmarks, likethis one by Alex Pliutau orthis one by Ruwan Fernando, suggests improvements of up to 10 times.

For chatty, distributed systems, these improvements accumulate quickly, making the difference not only noticeable in benchmarks, but also by the end-user.

API types

gRPC supports four different types of RPCs, unary, server streaming, client streaming, and bi-directional streaming. In reality, the messages are multiplexed using the same connection, but in the spirit of keeping things simple and approachable, this is not illustrated in the gRPC service model diagrams below.

Unary

Unary calls work the same way as a regular function call: a single request is sent to the server which in turn replies with a single response.

Server Streaming

In server streaming mode, the client sends a single request to the server, which in turn replies with multiple responses.

Client Streaming

The client streaming mode is the opposite of the server streaming mode. The client sends multiple requests to the server, which in turn replies with a single response.

Bi-directional streaming

In bi-directional streaming mode, both the client and the server may send multiple messages.

The `.proto` definition

The messages and services used for gRPC are described in .proto files, containingProtocol buffers, or protobuf, definitions.

The definition file is then used to generate code which can be used by both senders and receivers as a contract for communicating through these messages and services. As the binary format used by gRPC lacks any self-describing properties, this is the only way for senders and receivers to know how to interpret the messages.

Throughout this article, we'll use the hello.proto definition available for download on thek6 grpcbin website. For details on how to build your own grpc proto definition, see this excellent article from the official gRPC docs.

// ./definitions/hello.proto

// based on https://grpc.io/docs/guides/concepts.html

syntax = "proto2";

package hello;

service HelloService {
  rpc SayHello(HelloRequest) returns (HelloResponse);
  rpc LotsOfReplies(HelloRequest) returns (stream HelloResponse);
  rpc LotsOfGreetings(stream HelloRequest) returns (HelloResponse);
  rpc BidiHello(stream HelloRequest) returns (stream HelloResponse);
}

message HelloRequest {
  optional string greeting = 1;
}

message HelloResponse {
  required string reply = 1;
}

Getting started

With k6 v0.29.0, we're happy to introduce a native client for gRPC communication. In this early release, we've settled for providing a solid experience for unary calls. If any of the other modes would be particularly useful for you, we'd love to hear about your use case so we can move it up our backlog.

The current API for working with gRPC in k6 using the native client is as follows:

Method	Description
Client.load(importPaths, ...protoFiles)	Loads and parses the given protocol buffer definitions to be made available for RPC requests.
Client.connect(address [,params])	Opens a connection to the given gRPC server.
Client.invoke(url, request [,params])	Makes an unary RPC for the given service/method and returns a Response.
Client.close()	Close the connection to the gRPC service.

Creating the test

The gRPC module is a separate package, available from your test script as k6/net/grpc. Before we can use it, we first have to create an instance of the client. Instantiating the client, as well as the .load operation, is only available during test initialization, ie. directly in the global scope.

import grpc from 'k6/net/grpc';

const client = new grpc.Client();

Next, we'll load a .proto definition applicable for the system under test. For the purpose of this article, we'll use k6 grpcbin. Feel free to change this to whatever you please but keep in mind that you will also need an appropriate .proto definition for the server you're testing. The .load() function takes two arguments, the first one being an array of paths to search for proto files, and the second being the name of the file to load.

import grpc from 'k6/net/grpc';

const client = new grpc.Client();
client.load(['definitions'], 'hello.proto');

Once that is done, we'll go ahead and write our actual test.

import grpc from 'k6/net/grpc';

const client = new grpc.Client();
client.load(['definitions'], 'hello.proto');

export default () => {
  client.connect('grpcbin.test.k6.io:9001', {
    // plaintext: false
  });

  const data = { greeting: 'Bert' };
  const response = client.invoke('hello.HelloService/SayHello', data);

  check(response, {
    'status is OK': (r) => r && r.status === grpc.StatusOK,
  });

  console.log(JSON.stringify(response.message));

  client.close();
  sleep(1);
};

So let's walk through this script to make sure we understand what's happening. First, we use the.connect() function to connect to our system under test. By default, the client will set plaintext to false, only allowing you to use encrypted connections. If you, for any reason, need to connect to a server that lacks SSL/TLS, just flip this setting to true.

We then continue by creating the object we want to send to the remote procedure we're invoking. In the case of SayHello, it allows us to specify who the greeting should address using the greeting parameter.

Next, we invoke the remote procedure, using the syntax <package>.<service>/<procedure>, as described in our proto file. This call is made synchronously, with a default timeout of 60000 ms (60 seconds). To change the timeout, add the key timeout to the config object of .connect() with the duration as the value, for instance '2s' for 2 seconds.

Once we've received a response from the server, we'll then make sure the procedure executed successfully. The grpc module includes constants for this comparison which are listed here

Comparing the response status with grpc.StatusOK, which is 200 OK just like for HTTP/1.1 communication, ensures the call was completed successfully.

We'll then log the message in the response, close the client connection, and sleep for a second.

Executing the test

The test can be executed just like any other test, although you need to make sure you're on at least version v0.29.0 to have access to the gRPC module. To check this, run the following command:

$ k6 version
k6 v0.29.0 ((devel), go1.15.3, darwin/amd64)

Anything less than v0.29.0 here, will require you to first update k6. Instructions on how to do that can be found here.

Once that's out of the way, let's run our test:

$ k6 run test.js

          /\      |‾‾| /‾‾/   /‾‾/
     /\  /  \     |  |/  /   /  /
    /  \/    \    |     (   /   ‾‾\
   /          \   |  |\  \ |  (‾)  |
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: /Users/simme/code/grpc/test.js
     output: -

  scenarios: (100.00%) 1 scenario, 1 max VUs, 10m30s max duration (incl. graceful stop):
           * default: 1 iterations for each of 1 VUs (maxDuration: 10m0s, gracefulStop: 30s)

INFO[0000] {"reply":"hello Bert"} source=console

running (00m01.4s), 0/1 VUs, 1 complete and 0 interrupted iterations
default ✓ [======================================] 1 VUs 00m01.4s/10m0s 1/1 iters, 1 per VU

    ✓ status is OK

    checks...............: 100.00% ✓ 1 ✗ 0
    data_received........: 3.0 kB 2.1 kB/s
    data_sent............: 731 B 522 B/s
    grpc_req_duration....: avg=48.44ms min=48.44ms med=48.44ms max=48.44ms p(90)=48.44ms p(95)=48.44ms
    iteration_duration...: avg=1.37s min=1.37s med=1.37s max=1.37s p(90)=1.37s p(95)=1.37s
    iterations...........: 1 0.714536/s
    vus..................: 1 min=1 max=1
    vus_max..............: 1 min=1 max=1

From the output, we can now tell that our script is working and that the server indeed responds with a greeting addressed to who, or what, we supplied in our request body. We can also see that our check was successful, meaning the server responded with 200 OK.

Summary

In this article, we've gone through some of the fundamentals of gRPC and how it works. We've also had a look at the gRPC client introduced in k6 v0.29.0. Last, but not least, we've created a working test script demonstrating this functionality.

And that concludes this gRPC load testing tutorial. Thank you for reading!

How to Autoscale Kubernetes Pods with Keda - Testing with k6

Michael Wanyoike — Tue, 26 May 2020 18:22:23 +0000

Overview

This article demonstrates how to Load Test an application deployed in a Kubernetes cluster, verify that the autoscaling is working, and identify potential performance bottlenecks.

When you deploy an application to production on Kubernetes, you'll need to monitor traffic and scale up the application to meet heavy workloads during various times of the day. You'll also need to scale down your application to avoid incurring unnecessary costs. Doing this manually is impractical since heavy traffic can come at any time including late at night when your entire team is asleep. And even if you are awake, a spike might come and go before you had a chance to address it.

The best approach is to automate scaling. This can be triggered by monitoring custom metrics such cpu usage, network bandwidth or http requests per second. Scaling an application running on a Kubernetes platform can be done in the following ways:

Horizontal : Adjust the number of replicas(pods)
Vertical : Adjust resource requests and limits imposed on a container

In this article, we'll focus on horizontal scaling based on a custom metric. Which metric you should choose to trigger scaling is outside the scope of this article. We will use the http request rate metric in this example, however other metric types can be used. We'll cover how to obtain these metrics and visualize them for analysis. We'll use the following to perform this task:

k6 OSS: an open-source load testing tool. We'll use it to simulate heavy traffic(load) for our Kubernetes application. There's also Grafana Cloud k6 which you can use to scale load testing beyond your local computing and networking infrastructure limitations.
Prometheus: an open-source monitoring platform. We'll use it to scrape metrics from our application and the Kubernetes API in real-time as while the load testing tool is running.
Grafana: an open-source analytics platform. We'll use it to visualize real-time metrics being collected by Prometheus so that we can see the performance of our application across time. Below is a screenshot of how metrics are visualized on a Grafana dashboard.

The outline of this tutorial is as follows:

Application Deployment
Prometheus, Kube State Metrics and KEDA Deployment
InfluxDB and Grafana Setup
Configuring Horizontal Pod Autoscaling with Keda

You can follow along this entire tutorial on a single machine. Just note that results will be skewed since your CPU will handle both running the application and load testing it. For accurate results, these tasks need to be performed on separate machines. For learning and simplicity purposes, one machine will be sufficient.

The source code of the application we'll be load testing is provided in our GitHub repository. The load testing script is included there as well.

Below is a complete illustration of the entire setup:

Prerequisites

This article assumes you at least have some basic knowledge about running and configuring a Kubernetes cluster. We'll be using Minikube for this guide. However, feel free to use any other k8s implementation that can run on your computer.

Before we proceed, you'll need to have the following installed on your machine. For windows users, you can use Chocolatey package manager to install most of these requirements. For macOS, use brew. For Linux, use the instructions provided in the following links:

k6
Node.js
Docker
GNU Make (pre-installed for Linux, install xcode for macOS users)
kubectl
Minikube

Application Setup & Deployment

If you haven't, download the example project to your workspace now:



# Download project
git clone git@github.com:k6io/example-kubernetes-autoscaling-nodejs-api.git
cd example-kubernetes-autoscaling-nodejs-api

# Install dependencies
npm install

# Run development server
npm run dev

Open the URL http://localhost:4000/ to confirm the application is running. If you click on the crocodiles API link or visit the url http://localhost:4000/crocodiles, you should see the following:

Let's quickly inspect the code. Open the file db.json in your favorite code editor. This is where data is stored. You can add more records if you want. Next open server.js. This is the complete project code where the server logic lies. You'll see it's a simple project that uses json-server package to provide CRUD API services.

If you observe this section of code, you'll notice that an artificial delay has been implemented. Because the API application is so simple, we need to slow it down a bit just to help us simulate the results of a real-world application.



const minDelay = 30;
const maxDelay = 250;

// Add a delay to /crocodiles requests only
app.use('/crocodiles', function (req, res, next) {
  let delay = Math.floor(Math.random() * (maxDelay - minDelay)) + minDelay;
  setTimeout(next, delay);
});

The next section of the code I want to show you is how to export the metrics from our application to Prometheus. Later on, we will query these metrics to monitor the status of the application.



const prometheusExporter = require('@tailorbrands/node-exporter-prometheus');
const options = {
  appName: 'crocodile-api',
  collectDefaultMetrics: true,
  ignoredRoutes: ['/metrics', '/favicon.ico', '/__rules'],
};
const promExporter = prometheusExporter(options);
app.use(promExporter.middleware);
app.get('/metrics', promExporter.metrics);

If you would like, you can define additional metrics for your application. The above code will output several default metrics. In your browser, open the URL http://localhost:4000/metrics to view these metrics:

For this application, we are using the npm package @tailorbrands/node-exporter-prometheus to help us export metrics from our application in a format compatible with Prometheus scraping requirements with very little effort on our part. You can find other Prometheus client library for Node.js applications in the npmjs registry repository. If you are working with a different programming language, you can find official and non-official Prometheus client libraries on Prometheus website.

The metrics displayed in the above screenshot will update when you interact with application. Simply refreshing the URL http://localhost:4000/crocodiles in your browser or executing a curl command on the site will cause the values of the metrics to update. For example, the metric node_http_requests_total keeps track of the number of times a HTTP request is performed on the application. Do note the metrics page doesn't use AJAX. So you'll have to refresh from time to time to see the results.

When you setup Prometheus, it will fetch the values of this metrics every 5 - 15 seconds and store it in a time-series database. This is what is referred to as scraping. With this information, Prometheus can plot for you a graph so that you can see how the metric values change over time.

If you inspect the metrics provided by our application, you'll notice we have different metric types. For this guide, we'll focus on the metric node_http_requests_total which is of type counter. A counter is a cumulative metric that always keeps going up. It can be reset to 0 if the application is restarted.

Now, we run the k6 load testing tool to generate some traffic, and we will visualize how this counter metric changes over time. At the root of the application project, locate the script performance-script.js which contains instructions on how to perform the load test.

Below are 2 examples of the k6 load test configuration. The first option is a quick 3 minute load test you can use to quickly confirm metrics are being captured. The second option allows us to scale the number of virtual users over a duration of 12 minutes. This will give us enough data to analyze performance and behavior of our autoscaling configuration.



// First Load Testing Option : Quick Test with fixed no. of virtual users
export let options = {
  duration: '3m',
  vus: 200,
  thresholds: {
    http_req_duration: ['p(95)<700'],
  },
};

// Second Load Testing Option : Long Test with varying no. of virtual users
export let options = {
  stages: [
    { duration: '1m', target: 50 },
    { duration: '1m', target: 150 },
    { duration: '1m', target: 300 },
    { duration: '2m', target: 500 },
    { duration: '2m', target: 800 },
    { duration: '3m', target: 1200 },
    { duration: '3m', target: 50 },
  ],
};

Before we can run the load testing script, we need to deploy Prometheus to scrape the application's metric. Our application needs to be deployed as well in order for Prometheus to discover our application. To deploy our project to our Kubernetes(minikube) node, simply execute the following commands:



# Build the application's docker image inside the minikube's environment
make image
# Deploy the application
make apply

If you have trouble executing the above commands, just visit the file Makefile and execute the commands under image and apply sections. Below should be output of the last command:

Take note of the IP address and the port. In the above case, we can access our application through a web browser using the following address: http://10.98.55.109:4000. Unfortunately, the page will likely refuse to load since we haven't fully configured how our load balancer will expose our service. A quick work around is to create a tunnel by running the following command in a separate terminal:



minikube tunnel

Once the tunnel is up and running, you should be able to access the application in your browser. Proceed to the next section and deploy Prometheus.

Deploy Prometheus, Kube State Metrics and KEDA

To deploy Prometheus to our minikube node, follow this guide. You'll also need to deploy Kube State metrics. This is a service that accesses the Kubernetes API and provides Prometheus with metrics related to API objects such as deployments and pods. We need this service to track the number of running pods.

Since we are deploying, let's also deploy KEDA, a Kubernetes Event-driven Autoscaling service. It works alongside Horizontal Pod Autoscaler to scale pods up and down based on a threshold that we'll need to specify. Instructions for deploying using YAML files can be found on this page. For convenience, these are the commands you need to execute to deploy KEDA:



git clone https://github.com/kedacore/keda && cd keda

kubectl apply -f deploy/crds/keda.k8s.io_scaledobjects_crd.yaml
kubectl apply -f deploy/crds/keda.k8s.io_triggerauthentications_crd.yaml
kubectl apply -f deploy/

Use the command kubectl get po -A to ensure all the pods we have deployed are running. Use the command kubectl get service -n monitoring to find the ip address and port where Prometheus dashboard can be accessed. Construct the URL and launch the Prometheus dashboard in your browser. First let's confirm that Prometheus has discovered metrics from our application. From the top menu, go to Status > Targets page and scroll down to the bottom and look for the label crocodile-service:

If you see a similar status like the above, you are good to go on the next step. You should also confirm that the service kube-state-metrics has been discovered as well. On the top menu, click the Graph link to go to the graph page. This is where we enter query expressions to access the vast information that Prometheus is currently scraping.

Before we execute a query, fire up the k6 load testing tool to get some data to work with. You'll need to execute the load test script like this:



ENDPOINT=10.98.55.109:4000/crocodiles k6 run performance-test.js # replace ip address with yours

On the Graph page, enter this expression: node_http_requests_total. It should autofill for you as you type. Click on the Graph tab and you should see the following output.

This is how the results are visualized after the test has completed. Notice how it flattens at the top. This is because the metric stopped incrementing when the test ended. If you click on the Table tab, you should see two fields that look like this:

The node_http_requests_total metric keeps track of the total number of requests per HTTP status code. If you were to run the test, the line graph will start shooting up from where it left of. This metric doesn't seem useful in it's current form.

Fortunately, we can apply a function to make it useful. We can use the rate() function to calculate the number of requests per second over a specified duration. Update the expression as follows:

rate(node_http_requests_total[2m])

This function will give us the number of requests per second within a 2 minute window. Basically, it calculates how fast are the increments increasing per second. When the increments stop, the rate() function will give us 0. Below are the results of the load test with the rate function applied in the expression:

As you can see, our loading test script peaked at approximately 65 requests per second. When the load test script completed, the graph line goes back to 0. This is now a useful metric we can use to determine if scaling our pods is needed.

In the next section, we'll setup a more advance visualization dashboard that can display multiple metrics at once with automatic refresh rate.

Install InfluxDB and Grafana

In this section, we'll install InfluxDB which is an open-source time series database. InfluxDB is widely supported by many tools and applications. This provides new opportunities for collecting additional metric data from other sources such as Telegraf.

Grafana is an open-source analytic and monitoring platform that allows engineers to build dashboards for visually monitoring metrics from multiple sources in real-time. It supports the use of expressions to interpolate raw metric data into information that is easy to consume.

Both InfluxDB and Grafana can be deployed on the Kubernetes node. However, its easier and faster installing them locally. Do not use the Docker option as it's not easy for containers to communicating with Kubernetes services. Below are the download links:

Download InfluxDB : Get 1.8 OSS version
Download Grafana : Get latest version

Once you have installed both applications, make sure the services are running. For Ubuntu:



sudo systemctl start influxdb grafana-server

You can interact with influxDB database server via the influx command line interface. You can also interact with influxDB via http://localhost:8086 by passing query parameters.

Grafana is accessed by visiting http://localhost:3000. The default username and password should be admin admin. If this doesn't work, just edit the file /etc/grafana/grafana.ini and ensure the following lines are enabled:



[security]
# default admin user, created on startup
admin_user = admin

# default admin password, can be changed before first start of grafana,  or in profile settings
admin_password = admin

You'll need to restart grafana server for the the change to take effect. Once you log in, you'll need to go to Configuration > Data Sources and add the following sources:

InfluxDB K6 Database
Prometheus data source

Feel free to use other sources. In the screenshot provided, I've additionally installed Telegraf which is a service that collects real-time metrics from other database systems, IoT sensors and system performance metrics for CPU, memory, disk and network components. We won't use it for this article though.

Below are the settings that I've used for InfluxDB data source. The rest of the fields not mentioned here are left blank or in their default setting:

Name: InfluxDB-K6
URL: http://localhost:8086
Access: Server
Database: k6
HTTP Method: GET
Min time interval: 5s

Click the Save & Test button. If it says 'Data source is not working', it's because the database has not been created. When you run the following k6 command, the database will be created automatically:



# replace IP address
ENDPOINT=10.98.55.109:4000/crocodiles k6 run -o influxdb=http://localhost:8086/k6 performance-test.js

Below are the settings that I've used for Prometheus data source.

Name: InfluxDB-K6
URL: http://<prometheus ip address>:8080/
Access: Server
Scrape interval: 5s
Query timeout: 30s
HTTP Method: GET

Click Save & Test and make sure you get the message "Data source is working". Otherwise, you won't be able to proceed to the next step.

The next step is to create a new dashboard by importing the Crocodile Metrics Dashboard from this link. Copy and paste the JSON code and hit save. This custom dashboard will allow you to visually track:

HTTP Request Rate (sourced from both k6 and application via Prometheus)
Number of virtual users
Number of active application pods and their status
99th percentile response time (measured in milliseconds)
Memory usage per pod (measured in megabytes)

All the panels are easy to configure. You can change widget type, adjust the query and display more values. You can also re-organize the layout and add new panels. Below is how the Crocodile Metrics Dashboard should look like when installed:

In the next section, we'll configure KEDA to monitor and scale up our application.

Configuring Horizontal Pod Autoscaling with Keda

Before we configure auto scaling, let's run a quick load test. Open performance-test.js and ensure the following code is active:



export let options = {
  stages: [
    { duration: '1m', target: 50 },
    { duration: '1m', target: 150 },
    { duration: '1m', target: 300 },
    { duration: '2m', target: 500 },
    { duration: '2m', target: 800 },
    { duration: '3m', target: 1200 },
    { duration: '3m', target: 50 },
  ],
};

Run the k6 script using the following command ensuring InfluxDB is collecting K6 metrics.:



# replace IP address
ENDPOINT=10.98.55.109:4000/crocodiles k6 run -o influxdb=http://localhost:8086/k6 performance-test.js

The Grafana dashboard should start populating with data. As the number of http requests per second increases, the number of pods stays constant. You can wait for the load test to complete or you can cancel mid-way.

Let's now configure KEDA to monitor and scale our application. Open the YAML config file keda/keda-prometheus-scaledobject located in the project and analyze it:



apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
  name: prometheus-scaledobject
  namespace: default
  labels:
    deploymentName: crocodile-api
spec:
  scaleTargetRef:
    deploymentName: crocodile-api
  pollingInterval: 10 # Optional. Default: 30 seconds
  cooldownPeriod: 15 # Optional. Default: 300 seconds
  minReplicaCount: 1 # Optional. Default: 0
  maxReplicaCount: 10 # Optional. Default: 100
  triggers:
    - type: prometheus
      metadata:
        # Required
        serverAddress: http://10.103.240.12:8080
        metricName: access_frequency
        threshold: '50'
        query: sum(rate(node_http_requests_total[2m]))

Take note of the query we provided:

sum(rate(node_http_requests_total[2m]))

KEDA will run this query on Prometheus every 10 seconds. We've added the sum function to the expression in order to include data from all running pods. It will check the value against the threshold we provided, 50. If the value exceeds this amount, KEDA will increase the number of running pods up-to a maximum of 10. If the value is less, KEDA will scale our application pods back to 1. To deploy this configuration, execute the following command:



kubectl apply -f keda/keda-prometheus-scaledobject

Once again, run the k6 script test just like before and observe how the number of pods increases as the number of requests per second increase as well. Below is the final result after the test has completed.

Take note of the Running Pods chart. The number of application pods increased as the load increased. When the load throttled down, the number of pods decreased as well. And that's the beauty of automation.

Conclusion

How to Autoscale Kubernetes Pods with Keda - Testing with k6

How to autoscale pods using KEDA and Prometheus
How to analyse and visualize metrics using Grafana
How to extract metrics from applications and services

With the information you have at hand, you can benchmark any application and monitor the performance with accuracy. In addition to application metrics, you can also use:

CPU usage
Memory usage
Network usage
Disk usage
Database performance
Temperature

However, I would like to point out some important facts we haven't covered here that you need to learn:

For a real world application, you would have a separate pod containing the database. The data for this type of pod needs to synced when scaled out.
You will need to determine the request rate threshold that would make sense to scale

The more metrics you can monitor, the easier it is to identify bottlenecks affecting the performance of your application. For example, if your application has high CPU usage, optimizing the code can greatly boost the performance. If disk usage is high, using memory caching solutions can help a big deal.

The point is, scaling the number of application pods shouldn't be the only solution when it comes to handling heavy traffic. Scaling vertically or increasing the number of nodes should be considered and could greatly improve performance during heavy loads. By using k6 load testing tool, and using Grafana to analyze results, you can discover where bottlenecks for your application are located.

Forem: k6

Get Started with k6 browser

Testing Beyond Protocol Level

What is k6 browser module?

Writing the test

Running the Test

Browser Metrics

k6 Browser API

A Hybrid Approach to Performance Testing

Writing the test

Running the test

Get Involved

See Also

Testing the new AWS ARM CPU's with k6

The tests

Results (data transfer)

Results (CPU load)

Results

Finding .NET Memory Leaks through Soak Testing

.NET Garbage Collection 101

An Intro To GC

Talking ‘Bout My Generation: Making GC More Efficient

.NET Memory Leaks 101

What Are Memory Leaks In .NET and Why Do They Happen?

Why Should You Care About Memory Leaks?

Seeing a Memory Leak In Action

Running Soak Testing With k6

What is Soak Testing?

Getting Started With k6

Creating The Test Script

Running The Test Script

Analyzing The Results

Changing The Application’s Code

Running Soak Testing With k6, Again

Conclusion

How to Load Test Your Kafka Producers and Consumers using k6

xk6-kafka

Building k6 with the kafka extension

Running Kafka

k6 test

Import

Initialization

Default function

Teardown

Run the test

Scale the load

Extend the test

Kafka metrics in k6

Reader

Writer

More examples

Conclusion

How to Perform Load Testing with k6 using AWS CodeBuild Platform

📖What you will learn

Introduction

Writing your performance test

Creating the test script

Configuring the load

Configuring our thresholds

Setting up the CodeBuild workflow

Running k6 cloud tests

⚠️ Try it locally first

Scheduling Builds

Summary

How to Perform Load Testing with k6 using Google Cloud Build

📖 What you will learn

Introduction

Writing your performance test

Creating the test script

Configuring the load

Configuring our thresholds

Setting up the Cloud Build workflow

Running k6 Cloud tests

⚠️ Try it locally first

Nightly Builds

Summary

Benchmarking Redis with k6

redis-benchmark

Options

Examples