Forem: Ng Wai Foong

How to Load Test Your Kafka Producers and Consumers using k6

Ng Wai Foong — Wed, 22 Sep 2021 01:34:25 +0000

Recently, k6 started supporting k6 extensions to extend k6 capabilities for other cases required by the community. The community has already built plenty of extensions. k6 extensions are written in Go, and many of them are reusing existing Go libraries.

This makes k6 to be a versatile tool to test different protocols and adapt to multiple cases. This post is the third part of my series of articles testing various systems using k6:

Let's look in this post how we test the popular Kafka project. Apache Kafka is a powerful event streaming platform that provides the following features:

Write and read streams of events
Store streams of events for as long as you want to
Process streams of events in parallel retrospectively

It works by having client applications write events to the Kafka server. We term this type of application as Producers. Client applications which read and process events from the Kafka server are called Consumers.

Kafka itself is capable of handling hundreds to millions of events per second seamlessly on simple setup. But what if you wanted to test and observe how your Kafka service behaves before going live?

The xk6-kafka extension provides some convenient functions for interacting with Kafka Producers and Consumers. It serves as a producer that can send a high volume of messages per second, allowing you to monitor the system under test (SUT) and test how the application will keep up with the load.

xk6-kafka

At the time of this writing, the xk6-kafka extension provides the following APIs:

Function	Description
consume(reader, limit)	Consume messages from the Kafka server.
createTopic(address, topic)	Create a new topic.
listTopics(address)	Return a unique set of topics.
produce(writer, messages)	Produce messages to the Kafka server.
reader(brokers, topic)	Instantiate a new Reader instance.
writer(brokers, topic)	Instantiate a new Writer instance.

Some of the APIs mentioned above do accept additional optional parameters meant for authentication and message compression. Refer to more examples for additional information.

Building k6 with the kafka extension

By default, k6 does not support testing Kafka. Building k6 with the xk6-kafka extension creates a k6 version with the capabilities to test Kafka producers and consumers.

Make sure you have the following installed and ready before you continue:

Go (>=1.7)
Git

Next, continue the installation by running the following command in your terminal to install the xk6 module:

go install go.k6.io/xk6/cmd/xk6@latest

Once the command finishes successfully, you can start making your own custom k6 binary for Kafka as follows:

xk6 build --with github.com/mostafa/xk6-kafka@latest

It will take some time for the process to create a new k6 binary in your working directory.

Troubleshooting: If you experienced issues installing xk6 extension, kindly head over to the Github repository to download the pre-compiled binaries instead.

Running Kafka

The recommended approach is to use docker as manual installation is quite complicated and prone to errors. You can pull the following image by lensesio from DockerHub. It contains the complete Kafka setup for development.

docker pull lensesio/fast-data-dev:latest

After that, run the following to start the docker in detached mode:

sudo docker run -d --rm --name lenseio -p 2181:2181 -p 3030:3030 \
       -p 8081-8083:8081-8083 -p 9581-9585:9581-9585 -p 9092:9092  \
       -e ADV_HOST=127.0.0.1 lensesio/fast-data-dev

sudo docker logs -f -t lenseio

Visit http://localhost:3030 to get into the fast-data-dev environment.

k6 test

Import

Now, let’s create a new JavaScript file called test_script.js in the same directory as your k6 binary. Then, add the following import statement at the top of the file:

import { check } from "k6";
import { writer, produce, reader, consume, createTopic } from "k6/x/kafka";

Initialization

Continue by appending the following initialization code:

const bootstrapServers = ["localhost:9092"];
const kafkaTopic = "xk6_kafka_json_topic";

const producer = writer(bootstrapServers, kafkaTopic);
const consumer = reader(bootstrapServers, kafkaTopic);

The code will initialize both the writer and reader instances based on the configuration specified. If you are using a different IP/host address and port for your Kafka server, kindly modify it accordingly.

Next, call the createTopic function to create a new topic. Rest assured that this function will do nothing if the topic already exists.

createTopic(bootstrapServers[0], kafkaTopic);

Let’s create a function which generates a random integer as a unique identifier for each message later on. Please be noted that this is optional and not a mandatory requirement to do load testing.

function getRandomInt(max=1000) {
  return Math.floor((Math.random() * max) + 1);
}

Default function

As for the default function, define it as follows:

export default function () {
    let messages = [{
      key: JSON.stringify({
          correlationId: "test-id-sql-" + getRandomInt(),
      }),
      value: JSON.stringify({
          title: "Load Testing SQL Databases with k6",
          url: "https://k6.io/blog/load-testing-sql-databases-with-k6/",
          locale: "en"
      }),
    },
    {
      key: JSON.stringify({
          correlationId: "test-id-redis-" + getRandomInt(),
      }),
      value: JSON.stringify({
          title: "Benchmarking Redis with k6",
          url: "https://k6.io/blog/benchmarking-redis-with-k6/",
          locale: "en"
      }),
  }];

    let error = produce(producer, messages);
    check(error, {
          "is sent": (err) => err == undefined,
    });
}

The code block above works as follows:

Initialize a list of messages
Call produce function to publish the messages
Check if messages are successfully sent

Teardown

Once you are done with it, create a teardown function and close the connections:

export function teardown(data) {
    producer.close();
    consumer.close();
}

Run the test

Save the file and run the following command on your terminal:

./k6 run --vus 50 --duration 5s test_script.js

You should see the following output:

running (05.0s), 00/50 VUs, 15136 complete and 0 interrupted iterations
default ✓ [======================================] 50 VUs  5s

    ✓ is sent

    █ teardown

    checks.........................: 100.00% ✓ 15136  ✗ 0
    data_received..................: 0 B    0 B/s
    data_sent......................: 0 B    0 B/s
    iteration_duration.............: avg=16.49ms min=31.9µs med=13.52ms max=1.14s p(90)=28.55ms p(95)=36.46ms
    iterations.....................: 15136   3017.4609/s
    kafka.writer.dial.count........: 151    30.102841/s
    kafka.writer.error.count.......: 0      0/s
    kafka.writer.message.bytes.....: 5.2 MB  1.0 MB/s
    kafka.writer.message.count.....: 30272   6034.9218/s
    kafka.writer.rebalance.count...: 0      0/s
    kafka.writer.write.count.......: 30272   6034.9218/s
    vus............................: 5      min=5       max=50
    vus_max........................: 50     min=50      max=50

Scale the load

You can easily scale the load by increasing the number of vus. For example, the following command uses 500 vus to load test for a minute:

./k6 run --vus 500 --duration 1m test_script.js

If you are new to k6, check out how to configure the load options in the script or run a stress test with k6.

Extend the test

The script above is all about producing messages to your Kafka server. In fact, you can easily modify the code into a test which produces and consumes messages.

Simply add the following code below the for loop code:

let result = consume(consumer, 10);
check(result, {
    "10 messages returned": (msgs) => msgs.length == 10,
});

The code will read 10 messages each time. Simply modify the value to something higher if you wish to consume more messages.

The output is as follows when you run it with the same command:

running (05.0s), 00/50 VUs, 9778 complete and 0 interrupted iterations
default ✓ [======================================] 50 VUs  5s

    ✓ is sent
    ✓ 10 messages returned

    █ teardown

    checks.........................: 100.00% ✓ 19556      ✗ 0
    data_received..................: 0 B    0 B/s
    data_sent......................: 0 B    0 B/s
    iteration_duration.............: avg=25.53ms min=41.4µs med=18ms max=1.41s p(90)=37.73ms p(95)=52.37ms
    iterations.....................: 9778   1946.80798/s
    kafka.reader.dial.count........: 50     9.955042/s
    kafka.reader.error.count.......: 0      0/s
    kafka.reader.fetches.count.....: 101    20.109184/s
    kafka.reader.message.bytes.....: 15 MB   2.9 MB/s
    kafka.reader.message.count.....: 97830   19478.034846/s
    kafka.reader.rebalance.count...: 0      0/s
    kafka.reader.timeouts.count....: 46     9.158638/s
    kafka.writer.dial.count........: 152    30.263327/s
    kafka.writer.error.count.......: 0      0/s
    kafka.writer.message.bytes.....: 3.4 MB  669 kB/s
    kafka.writer.message.count.....: 19556   3893.615961/s
    kafka.writer.rebalance.count...: 0      0/s
    kafka.writer.write.count.......: 19556   3893.615961/s
    vus............................: 50     min=50      max=50
    vus_max........................: 50     min=50      max=50

Kafka metrics in k6

By default, k6 has its own built-in metrics which are collected automatically. Apart from that, you can create your own custom metrics. Custom metrics can be categorized into the following types:

Counter: A metric that cumulatively sums added values.
Gauge: A metric that stores the min, max and last values added to it.
Rate: A metric that tracks the percentage of added values that are non-zero.
Trend: A metric that allows for calculating statistics on the added values (min, max, average and percentiles).

Besides k6, k6 extensions can collect metrics and report them as part of the k6 results output. In this case, xk6-kafka collects individual stats for both reader and writer.

Reader

Let’s have a look at the metrics meant for the reader.

Metrics	Type	Description
kafka.reader.dial.count	Counter	Total number of times the reader tries to connect to Kafka.
kafka.reader.error.count	Counter	Total number of errors occurred when reading from Kafka.
kafka.reader.fetches.count	Counter	Total number of times the reader fetches batches of messages from Kafka.
kafka.reader.message.bytes	Counter	Total bytes consumed.
kafka.reader.message.count	Counter	Total number of messages consumed.
kafka.reader.rebalance.count	Counter	Total number of rebalances of a topic in a consumer group (deprecated) .
kafka.reader.timeouts.count	Counter	Total number of timeouts occurred when reading from Kafka

Writer

As for the writer, the metrics are as follows:

Metrics	Type	Description
kafka.writer.dial.count	Counter	Total number of times the writer tries to connect to Kafka.
kafka.writer.error.count	Counter	Total number of errors occurred when writing to Kafka.
kafka.writer.message.bytes	Counter	Total bytes produced.
kafka.writer.message.count	Counter	Total number of messages produced.
kafka.writer.rebalance.count	Counter	Total number of rebalances of a topic (deprecated).
kafka.writer.write.count	Counter	Total number of times the writer writes batches of messages to Kafka.

There are more available kafka metrics, as you can find them here. However, the extension does not collect all metrics yet. You can follow this GitHub issue to track the progress of their additions.

More examples

Moreover, the xk6-kafka repository provides a few test scripts that work out-of-the-box for new users. At the time of this writing, it comes with the following tests:

test_avro.js : Tests Kafka with 200 Avro messages per iteration.
test_avro_with_schema_registry.js : Tests Kafka with 200 Avro messages per iteration using schema registry
test_json.js : Tests Kafka with 200 JSON messages per iteration.
test_json_with_snappy_compression.js : Tests Kafka with 200 JSON messages per iteration using snappy compression
test_sasl_auth.js : Tests Kafka with 200 JSON messages per iteration and based on SASL authentication.
test_topics.js : List topics on all Kafka partitions and create random topics.

Feel free to experiment with them and modify the code accordingly based on your own use cases. If you bump into issues, report them on GitHub.

Conclusion

In conclusion, load testing Apache Kafka is now a lot easier with k6. k6 provides the foundation to create and scale your load tests, and the xk6-kafka extension brings a convenient API to interact with a Kafka server.

If you wish to find out more on other available k6 extensions, simply head over to the bundle builder page. The page also allows you to generate the corresponding command for building your own custom k6 binary.

If you have any questions or are interested in building an extension, join the k6 community on Slack.

Benchmarking Redis with k6

Ng Wai Foong — Wed, 18 Aug 2021 01:47:15 +0000

Previously, I have covered an article on Load Testing SQL Databases with k6. For your information, from k6 version 0.29.0 onwards, you can write a k6 Go extension and build your own k6 binaries. This comes in handy as you can use a single framework for load testing different protocols, such as ZMTQ, SQL, Avro, MLLP, etc.

In this series of k6 extensions, let’s benchmark Redis now. According to redis.io, Redis is a type of in-memory data structure store that can be used as database, cache and message broker.

You might want to evaluate the performance or scalability of Redis instances in given hardware, giving you better insights into the throughput the Redis service can handle.

This tutorial covers Redis performance testing via two different approaches on a Linux machine:

redis-benchmark
xk6-redis

redis-benchmark

By default, Redis comes with its own benchmark utility called redis-benchmark. It is similar to Apache's ab utility and can simulate a number of clients sending a total number of queries simultaneously.

Options

Make sure that you have Redis installed in your system. If you have not done so, kindly head over to the official Redis download page and install it based on the instructions given.

Once you are done with it, you should be able to run the following command:

redis-benchmark --help

You should see the following output:

Usage: redis-benchmark [-h <host>] [-p <port>] [-c <clients>] [-n <requests>] [-k <boolean>]

 -h <hostname>      Server hostname (default 127.0.0.1)
 -p <port>          Server port (default 6379)
 -s <socket>        Server socket (overrides host and port)
 -a <password>      Password for Redis Auth
 -c <clients>       Number of parallel connections (default 50)
 -n <requests>      Total number of requests (default 100000)
 -d <size>          Data size of SET/GET value in bytes (default 3)
 --dbnum <db>       SELECT the specified db number (default 0)
 -k <boolean>       1=keep alive 0=reconnect (default 1)
 -r <keyspacelen>   Use random keys for SET/GET/INCR, random values for SADD
  Using this option the benchmark will expand the string __rand_int__
  inside an argument with a 12 digits number in the specified range
  from 0 to keyspacelen-1. The substitution changes every time a command
  is executed. Default tests use this to hit random keys in the
  specified range.
 -P <numreq>        Pipeline <numreq> requests. Default 1 (no pipeline).
 -e                 If server replies with errors, show them on stdout.
                    (no more than 1 error per second is displayed)
 -q                 Quiet. Just show query/sec values
 --csv              Output in CSV format
 -l                 Loop. Run the tests forever
 -t <tests>         Only run the comma separated list of tests. The test
                    names are the same as the ones produced as output.
 -I                 Idle mode. Just open N idle connections and wait.

Examples

Depending on your needs, a typical example is to just run the benchmark with the default configuration:

redis-benchmark

It is a good idea to use the -q option. Here is an example for running 100k of requests in quiet mode:

redis-benchmark -q -n 100000

In addition, you can run parallel clients via the -c option. The following example use 20 parallel clients for a total of 100k requests:

redis-benchmark -q -n 100000 -c 20

You can restrict the test to run only a subset of the commands. For example, you can use the following command to test only set and get commands:

redis-benchmark -q -t set,get -n 100000

In fact, you can run test on specific commands for benchmarking like the following example:

redis-benchmark -q -n 100000 script load "redis.call('set','key','value')"

If your Redis server is running on a different hostname and port, you can benchmark the server as follows:

redis-benchmark -h 192.168.1.1 -p 6379 -n 100000 -c 20

You should get the following output indicating the requests per second for each of the test conducted:

PING_INLINE: 43478.26 requests per second
PING_BULK: 41666.67 requests per second
SET: 43478.26 requests per second
GET: 43478.26 requests per second
INCR: 40000.00 requests per second
LPUSH: 43478.26 requests per second
RPUSH: 37037.04 requests per second
LPOP: 45454.55 requests per second
RPOP: 34482.76 requests per second
SADD: 43478.26 requests per second
HSET: 45454.55 requests per second
SPOP: 45454.55 requests per second
LPUSH (needed to benchmark LRANGE): 40000.00 requests per second
LRANGE_100 (first 100 elements): 45454.55 requests per second
LRANGE_300 (first 300 elements): 43478.26 requests per second
LRANGE_500 (first 450 elements): 47619.05 requests per second
LRANGE_600 (first 600 elements): 38461.54 requests per second
MSET (10 keys): 41666.67 requests per second

Latency

Sometimes, you might prefer to analyze the latency instead. There are two types of latency measurement provided by redis-cli:

latency
intrinsic latency

In this case, we measure latency as the time between sending a request to Redis and receiving a response. On the other hand, intrinsic latency refers to the system latency that is highly dependent on external factors such as operating system kernel or virtualization. Since Redis 2.8.7, you can measure the intrinsic latency independently.

Please note that you can only run redis-cli in the machine which hosts the Redis server unlike redis-benchmark which is runnable on the client machine. Besides that, this mode is not connected to a Redis server at all and the measurement is based on the largest time in which the kernel does not provide CPU time to run to the redis-cli process itself. As a result, it is not an actual measurement of the latency between client and Redis server.

Having said that, it does provide a quick analysis if there is something wrong with the machine that hosts the Redis server.

Run the following command to get the overall latency of your Redis server:

redis-cli --latency

You should see an increase in the sample as time goes by and the average latency:

min: 0, max: 5, avg: 0.22 (2406 samples)

Use Ctrl+C to stop it as the process will run indefinitely without stopping.

For intrinsic latency, you should use the following command instead:

redis-cli --intrinsic-latency 10

You can pass an integer representing the duration of the test. In this case, the test will run for 10 seconds. The output is as follows:

Max latency so far: 1 microseconds.
Max latency so far: 15 microseconds.
Max latency so far: 16 microseconds.
Max latency so far: 17 microseconds.
Max latency so far: 18 microseconds.
Max latency so far: 20 microseconds.
Max latency so far: 21 microseconds.
Max latency so far: 24 microseconds.
Max latency so far: 25 microseconds.
Max latency so far: 50 microseconds.
Max latency so far: 74 microseconds.
Max latency so far: 87 microseconds.
Max latency so far: 150 microseconds.
Max latency so far: 1089 microseconds.
Max latency so far: 1715 microseconds.
Max latency so far: 2344 microseconds.
Max latency so far: 7438 microseconds.
Max latency so far: 8002 microseconds.

158645097 total runs (avg latency: 0.0630 microseconds / 63.03 nanoseconds per run).
Worst run took 126948x longer than the average latency.

The average latency is about 0.22 milliseconds while the intrinsic latency is 0.063 microseconds.

Let’s proceed to the next section and start exploring another testing approach using k6.

xk6-redis

k6 provides the capabilities to do performance testing with scripting language. This is a big plus to developers and Q&A testers as you will have a better control of the entire workflow of the test. For example, you can ramp up or ramp down the requests at specific intervals of the test which is not achievable when using redis-benchmark.

Fortunately, k6 provides the xk6-redis extension as part of their ecosystem. You can use it directly to build your own custom k6 binaries for testing Redis server.

This extension comes with the following API:

Output	Usage
Client(options)	Represent the Client construtor. Returns a new Redis client object.
client.set(key, value, expiration time)	Set the given key with the given value and expiration time.
client.get(key)	Get returns the value for the given key.

Building k6 with the redis extension

Before that, make sure you have the following installed in your machine:

Once you have completed the installation, run the following to install xk6 module:

go install github.com/k6io/xk6/cmd/xk6@latest

If you have installed xk6 directory to Go module, you can make your Redis k6 build by running:

xk6 build --with github.com/k6io/xk6-redis

You should get a k6 executable in your current working directory.

Alternatively, you can download the pre-compiled binaries at the following Github repository. The latest version at the time of this writing is v0.4.1. If you have trouble identifying the architecture of your Linux machine, simply run the following command:

dpkg --print-architecture

Let’s say that the command returns the following:

amd64

You should download the xk6_0.4.1_linux_amd64.tar.gz asset and extract it as follows:

tar -xvf xk6_0.4.1_linux_amd64.tar.gz

You should get the following files in your working directory:

README.md
LICENSE
xk6

Then, run the following command to build k6 for Redis:

./xk6 build --with github.com/k6io/xk6-redis

You should have now a new k6 binary in your working directory.

k6 Script

Next, let’s create a new JavaScript file called test_script.js in the same directory as your k6 executable. Append the following import statement at the top of the file:

import redis from 'k6/x/redis';

Continue by adding the following code which connect to your Redis server:

const client = new redis.Client({
  addr: 'localhost:6379',
  password: '',
  db: 0,
});

It accepts the following an object with the following fields:

addr: hostname and port of your Redis server denoted as hostname:port.
password: password of your Redis server.
db: the db number ranging from 0 to 15.

To keep it simple and short, the test case is going to be as follows:

Set a new key:value on start of test.
Running parallel VUs to get the same key repeatedly.

The k6 setup function runs only once at the test start, independently of the test load and duration. Let’s set the key: value as follows:

export function setup() {
  client.set('key', 'value', 0);
}

The set function accepts three input parameters:

key
value
expiration time

Then, define the default function which will be called repeatedly by each VU during the entire test:

export default function () {
  client.get('key');
}

The complete code is as follows:

import redis from 'k6/x/redis';
import { check } from 'k6';

const client = new redis.Client({
  addr: 'localhost:6379',
  password: '',
  db: 0,
});

export function setup() {
  client.set('key', 'value', 0);
}

export default function () {
  client.get('key');
}

Running the test

Save the test script and run the following command to the test your Redis server for 5 seconds:

./k6 run test_script.js --duration 5s

By default, it is using one Virtual User (VU) but you can modify it with the --vus flag. You should see the following output:

          /\      |‾‾| /‾‾/   /‾‾/   
     /\  /  \     |  |/  /   /  /    
    /  \/    \    |     (   /   ‾‾\  
   /          \   |  |\  \ |  (‾)  | 
  / __________ \  |__| \__\ \_____/ .io

  execution: local
    script: test_script.js
    output: -

  scenarios: (100.00%) 1 scenario, 1 max VUs, 35s max duration (incl. graceful stop):
        * default: 1 looping VUs for 5s (gracefulStop: 30s)


running (05.0s), 0/1 VUs, 42037 complete and 0 interrupted iterations
default ✓ [======================================] 1 VUs  5s

    █ setup

    data_received........: 0 B   0 B/s
    data_sent............: 0 B   0 B/s
    iteration_duration...: avg=104.45µs min=53.7µs med=88.6µs max=9.32ms p(90)=115.4µs p(95)=129.5µs
    iterations...........: 42037 8401.691798/s
    vus..................: 1    min=1       max=1
    vus_max..............: 1    min=1       max=1

This test reports that the Redis server handles 8401 iterations per second. Because each iteration refers to one execution of the default function and there is one request call in our default function, the server is handling 8401 GET requests per second in this test.

Scale the load

Let’s increase the load gradually until it encounters an error. For a start, set the VUs to 100 as follows:

./k6 run test_script.js --duration 5s --vus 100

The output is as follows:

running (05.0s), 000/100 VUs, 111939 complete and 0 interrupted iterations
default ↓ [======================================] 100 VUs  5s

    █ setup

    data_received........: 0 B  0 B/s
    data_sent............: 0 B  0 B/s
    iteration_duration...: avg=4.39ms min=46.8µs med=3.32ms max=87.24ms p(90)=9.5ms p(95)=12.51ms
    iterations...........: 111939 22304.954101/s
    vus..................: 100  min=100     max=100
    vus_max..............: 100  min=100     max=100

It indicates that your Redis server can sustain about 22304 iterations per second for 100 users at the same time.

Continue the test and set the VUs to 1000 this time:

./k6 run test_script.js --duration 5s --vus 1000

Depending on the configuration of your Redis, you might encounter the following error:

ERRO[0003] ERR max number of clients reached
running at go.k6.io/k6/js/common.Bind.func1 (native)
default at file:///home/wfng/test_script.js:14:14(4)  executor=constant-vus scenario=default source=stacktrace

It indicates that you have reached the max number of clients allowed. You can check the number of active connection by running the following command inside redis-cli:

info clients

It will returns the following output:

# Clients
connected_clients:7
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

To get the max limit, use the following instead:

config get maxclients

The output is as follows:

1) "maxclients"
2) "500"

Latency

Now, let’s have a look at how to get the latency via k6. At the time of this writing, the xk6-redis extension does not report latency as part of its metrics. However, you can easily extend the code in your script and implement your own custom metrics.

Have a look at the following workaround to measure latency. First, let’s add the following import statement at the top of your k6 script:

import { Trend } from 'k6/metrics';

Then, initialize a Trend instance as follows:

let RedisLatencyMetric = new Trend('redis_latency', true);

It accepts two input arguments:

name: the name of the custom metric.
isTime: a boolean indicating whether the values added to the metric are time values or just untyped values.

Add the final touch by modifying the default function as follows:

export default function () {
  const start = Date.now();
  client.get('key');
  const latency = Date.now() - start;
  RedisLatencyMetric.add(latency);
}

Have a look at the following complete code which initialize the options directly inside the script:

import { Trend } from 'k6/metrics';
import redis from 'k6/x/redis';

let RedisLatencyMetric = new Trend('redis_latency', true);

export let options = {
  vus: 40,
  duration: '10s',
}

const client = new redis.Client({
  addr: 'localhost:6379',
  password: '',
  db: 0,
});

export function setup() {
  client.set('key', 'value', 0);
}

export default function () {
  const start = Date.now();
  client.get('key');
  const latency = Date.now() - start;
  RedisLatencyMetric.add(latency);
}

You should be able to see redis_latency metrics once the test has completed.

iteration_duration...: avg=782.57µs min=67.35µs med=732.92µs max=15.86ms p(90)=1.1ms p(95)=1.3ms
iterations...........: 506755 50660.636169/s
redis_latency........: avg=764.8µs  min=0s     med=1ms     max=16ms    p(90)=1ms   p(95)=1ms

⚠️ Please note that this workaround of measuring the latency is only indicative, as the JavaScript implementation adds an overhead that might skew the reported latency, especially when the latency is in the sub-microsecond range.

It would be great if the xk6-redis extension provided its own built-in Redis latency metrics similar to the HTTP request metrics. Measuring Redis latency in Go directly would be much more accurate and avoid the unnecessary RedisLatencyMetric script code.

Conclusion

All in all, redis-benchmark is a good tool that provides you with a quick glimpse of the performance of your Redis server. On the other hand, k6 is scriptable in JavaScript and can provide you with better control over the execution and workflow of your test. A scripting language is more flexible for testing various ways to connect and query your Redis server.

In fact, you can utilize both of the tools to get the best out of them. For example, you can run redis-benchmark when you install it on your machine for the first time, to get a rough idea of the performance. Subsequently, use k6 for more advanced cases like integrating your test with your existing toolbox or automating your testing.

Load Testing SQL Databases with k6

Ng Wai Foong — Fri, 16 Jul 2021 03:46:34 +0000

This short tutorial shows how to run a k6 test for load testing a database.

In performance testing, we often trigger load tests that simulate realistic user flows, particularly those that are most commonly seen in production. This type of acceptance testing usually interacts with various parts of our infrastructure: web servers, microservices, databases, etc.

But what if you want to test the performance or scalability of an infrastructure resource in isolation?

In many cases, internal components use custom protocols, and the testing tool needs to support those protocols to test the resource individually. Luckily, with k6, you can use or create extensions that allow you to test different protocols, such as ZMTQ, SQL, Avro, MLLP, etc.

One of the components that you might want to test separately is the database. Databases play an essential role in the performance of our applications, and they can be the bottleneck when experiencing a high volume of users.

Load testing the database directly could provide you with better insights about the database performance in advance. As a result, you could thoroughly plan out your database architecture and determine how to scale it properly.

In this tutorial, let’s explore how to load test a database using the xk6-sql extension. For simplicity, the tests will be executed against a local SQLite3 server but the extension supports the following RDBMS databases:

PostgreSQL
MySQL
SQLite3
MS SQL

Build

In this section, you will install all the required components and build a k6 binary for SQL.

Install the C Compiler for SQLite3

A C compiler is required if you are using SQLite3. Simply install the build-essential package if you are using Debian-based operating system. For Windows users, download the tdm-gcc compiler, extract it and place it in any directory that you prefer. Then, add the path of the bin folder to Environment Variable as follows:

Install the Golang Toolchain

Head over to Golang’s installation page and download the installer based on the operating system of your machine. Once you have installed, run the following command to verify the version.



go version

You should get information related the version number of Go as well as your system architecture:



go version go1.16.4 windows/amd64

Build the k6 binary including the SQL extension

For non-SQLite database, run the following command to build the k6 binary:



xk6 build master --with github.com/imiric/xk6-sql

You need to set CGO_ENABLED to 1 when building for SQLite3 to ensure that C compiler is used:



CGO_ENABLED=1 xk6 build master --with github.com/imiric/xk6-sql

On Windows platform, you need set it explicitly, using set first to call the build command:



set CGO_ENABLED=1
xk6 build master --with github.com/imiric/xk6-sql

You should see the following output on your console:



2021/06/17 14:29:43 [INFO] Temporary folder: C:\Users\wfng\AppData\Local\Temp\buildenv_2021-06-17-1429.359000039
2021/06/17 14:29:43 [INFO] Writing main module: C:\Users\wfng\AppData\Local\Temp\buildenv_2021-06-17-1429.359000039\main.go
2021/06/17 14:29:43 [INFO] Initializing Go module
2021/06/17 14:29:43 [INFO] exec (timeout=10s): C:\Program Files\Go\bin\go.exe mod init k6
go: creating new go.mod: module k6
go: to add module requirements and sums:
        go mod tidy
2021/06/17 14:29:44 [INFO] Pinning versions
2021/06/17 14:29:44 [INFO] exec (timeout=0s): C:\Program Files\Go\bin\go.exe get -d -v go.k6.io/k6@master
go: downloading go.k6.io/k6 v0.32.1-0.20210616133500-9f3dd60fbdc1
go get: added go.k6.io/k6 v0.32.1-0.20210616133500-9f3dd60fbdc1
2021/06/17 14:30:50 [INFO] exec (timeout=0s): C:\Program Files\Go\bin\go.exe get -d -v github.com/imiric/xk6-sql
go get: added github.com/imiric/xk6-sql v0.0.0-20210517160107-d222ad8b93eb
2021/06/17 14:30:52 [INFO] Build environment ready
2021/06/17 14:30:52 [INFO] Building k6
2021/06/17 14:30:52 [INFO] exec (timeout=0s): C:\Program Files\Go\bin\go.exe mod tidy
2021/06/17 14:30:56 [INFO] exec (timeout=0s): C:\Program Files\Go\bin\go.exe build -o C:\Users\wfng\Documents\k6_test\k6.exe -ldflags -w -s -trimpath
2021/06/17 14:31:15 [INFO] Build complete: .\k6.exe
2021/06/17 14:31:15 [INFO] Cleaning up temporary folder: C:\Users\wfng\AppData\Local\Temp\buildenv_2021-06-17-1429.359000039

After that, you should have a new k6 binary in your working directory. Since I am building on the Windows platform, I got the k6.exe executable file.

k6 Script

You need to write a JavaScript file in order to perform load testing with k6. Let’s have a look at an example of a simple test script for load testing API via HTTP:



import http from 'k6/http';
import { sleep } from 'k6';

export default function () {
  http.get('https://test.k6.io');
  sleep(1);
}

Each test script requires a default function which will be executed over and over again during testing. The script above makes a GET call to our own k6 test API and sleeps for a second on each execution for a single VU.

For load testing a database, all you need to do is to import the SQL module that you have created earlier and write the corresponding code inside the default function.

Create a new JavaScript file called script.js in the same directory as your k6 binary file.

Import SQL module

You can import your newly created SQL module by adding this line to script.js:



import sql from 'k6/x/sql';

The naming is based on what has been defined in the Go file. In this case, it is defined as k6/x/sql.

Connect to database

You can easily connect to your database by calling the sql.open function:



const db = sql.open("sqlite3", "./test.db");

It accepts two input parameters:

type - the type of database (mysql, postgres, sqlite3, sqlserver)
name - the name of the database

Setup and Teardown the database

Before executing a SQL command, let’s explore a little more about the k6 test life cycle. It typically follows this structure:



// 1. init code (call once per VU)

export function setup() {
  // 2. setup code (call once at the beginning of test)
}

export default function (data) {
  // 3. VU code
}

export function teardown(data) {
  // 4. teardown code (call once at the end of test)
}

You can add in any init code right before the setup, default function and teardown. Init code serves as the initialization and will be called once for each virtual user (VU).

Also, you can specify a setup function which is called once at the beginning of the test where VU is 0. On the other hand, teardown is called once at the end of the test.

As explained earlier, the default function serves as the VU code which will be executed continuously during testing.

Execute SQL command

After connecting to the database, you can use the db object and call the exec to run any SQL command.

For example, as part of the setup process, before the “load” runs, you can create a new table and insert a few rows of data into the table as follows:



export function setup() {
  db.exec(`CREATE TABLE IF NOT EXISTS person (
           id integer PRIMARY KEY AUTOINCREMENT,
           email varchar NOT NULL,
           first_name varchar,
           last_name varchar);`);

  db.exec("INSERT INTO person (email, first_name, last_name) VALUES('johndoe@email.com', 'John', 'Doe');");
  db.exec("INSERT INTO person (email, first_name, last_name) VALUES('marysue@email.com', 'Mary', 'Sue');");
  db.exec("INSERT INTO person (email, first_name, last_name) VALUES('dorydoe@email.com', 'Dory', 'Doe');");
}

And you should not forget to clean up the database at the end of the test with the teardown function. This example deletes the table and closes the database connection:



export function teardown() {
  db.exec("DELETE FROM person;");
  db.exec("DROP TABLE person;");
  db.close();
}

Query data from database

You can easily query the output with the query function. Let’s use it as part of load testing to determine how many iterations you can get when querying the database:



export default function () {
  let results = sql.query(db, "SELECT * FROM person;");
}

As usual, you can run the check statement to determine the output. Let’s do a simple check on the total rows of data in the database:



import { check } from 'k6';

...

export default function () {
  let results = sql.query(db, "SELECT * FROM person;");
  check(results, {
    'is length 3': (r) => r.length === 3,
  });
}

The complete script code is as follows:



import sql from 'k6/x/sql';
import { check } from 'k6';

const db = sql.open("sqlite3", "./test.db");

export function setup() {
  db.exec(`CREATE TABLE IF NOT EXISTS person (
           id integer PRIMARY KEY AUTOINCREMENT,
           email varchar NOT NULL,
           first_name varchar,
           last_name varchar);`);

  db.exec("INSERT INTO person (email, first_name, last_name) VALUES('johndoe@email.com', 'John', 'Doe');");
  db.exec("INSERT INTO person (email, first_name, last_name) VALUES('marysue@email.com', 'Mary', 'Sue');");
  db.exec("INSERT INTO person (email, first_name, last_name) VALUES('dorydoe@email.com', 'Dory', 'Doe');");
}

export function teardown() {
  db.exec("DELETE FROM person;");
  db.exec("DROP TABLE person;");
  db.close();
}

export default function () {
  let results = sql.query(db, "SELECT * FROM person;");
  check(results, {
    'is length 3': (r) => r.length === 3,
  });
}

Running the test

Once we have the completed script, you can run the test. Let’s start running the load test for 5 seconds:



k6 run script.js --duration 5s

By default, it is using just one virtual user (VU) but you can modify it via --vus flag. You should see the following output:



          /\      |‾‾| /‾‾/   /‾‾/
     /\  /  \     |  |/  /   /  /
    /  \/    \    |     (   /   ‾‾\
   /          \   |  |\  \ |  (‾)  |
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: script.js
     output: -

  scenarios: (100.00%) 1 scenario, 1 max VUs, 35s max duration (incl. graceful stop):
           * default: 1 looping VUs for 5s (gracefulStop: 30s)


running (05.1s), 0/1 VUs, 34467 complete and 0 interrupted iterations
default ✓ [======================================] 1 VUs  5s

     ✓ is length 3

     █ setup

     █ teardown

     checks...............: 100.00% ✓ 34467       ✗ 0
     data_received........: 0 B     0 B/s
     data_sent............: 0 B     0 B/s
     iteration_duration...: avg=143.57µs min=0s med=0s max=43.24ms p(90)=519.2µs p(95)=985.47µs
     iterations...........: 34467   6812.032587/s
     vus..................: 1       min=1         max=1
     vus_max..............: 1       min=1         max=1

In this case, it shows that the database can handle about 6812 queries per second and average time of 144µs per iteration.

Scale the load

In the previous test, you have specified just a single virtual user. Let’s scale it up to 10 and see how SQLite performs. Run the following command:



k6 run script.js --duration 5s --vus 10

You should get the following result:



running (05.1s), 00/10 VUs, 43228 complete and 0 interrupted iterations
default ✓ [======================================] 10 VUs  5s

    ✓ is length 3

    █ setup

    █ teardown

    checks...............: 100.00% ✓ 43228    ✗ 0
    data_received........: 0 B  0 B/s
    data_sent............: 0 B  0 B/s
    iteration_duration...: avg=1.16ms min=0s med=0s max=136.03ms p(90)=522.5µs p(95)=570.15µs
    iterations...........: 43228   8446.461494/s
    vus..................: 10   min=10      max=10
    vus_max..............: 10   min=10      max=10

Let’s continue the test and set the VU to 100 this time.



k6 run script.js --duration 5s --vus 100

The output is as follows:



default ✓ [======================================] 100 VUs  5s

    ✓ is length 3

    █ setup

    █ teardown

    checks...............: 100.00% ✓ 97490        ✗ 0
    data_received........: 0 B  0 B/s
    data_sent............: 0 B  0 B/s
    iteration_duration...: avg=5.07ms min=0s med=506.55µs max=140.07ms p(90)=18.13ms p(95)=28.58ms
    iterations...........: 97490   19034.709634/s
    vus..................: 100  min=100     max=100
    vus_max..............: 100  min=100     max=100

It indicates that SQLite is capable of supporting 100 users with an average duration of 5.07ms per transaction.

For actual use cases, you should continue to scale it up to the point where it will congest your database and cause it to breakdown. This allows you to have a better idea on the maximum limit of your database.

If you are new to k6, check out how to configure the load options in the script or run a stress test with k6.

About k6 Extensions

For your information, you can combine multiple extensions and build your own custom k6 binary. For example, you can use the following command to build a k6 binary for both sql and redis:



xk6 build v0.32.0 --with github.com/dgzlopes/xk6-redis --with github.com/imiric/xk6-sql

Simply head over to the bundle builder page to generate the corresponding command based on your own use cases.

With this tutorial, I wanted to show you how easy it is to load test your database or other dependencies separately with k6. If you have any questions or are interested in building an extension, join the k6 community on Slack.

What’s New in Rasa 2.0 - Build Your Own Chatbot

Ng Wai Foong — Sun, 11 Oct 2020 08:10:00 +0000

The following post is a simplified version of my original article that was published on Medium.

Introduction

Rasa Open Source is a machine learning framework used to build text- and voice-based chatbots. Recently, it released their first official version 2.0 on October 2020. The new version aims at unifying training data formats, configuration files, and the way to handle models. As a result, there are quite a number of major breaking changes compared to the previous version.

In this post, we are going to dive deep into some of the major differences between Rasa 1.10 and the latest version 2.0. You should check it out and determine if it is worth the effort before performing the migration for your old Rasa server.

Folder and Files Hierarchy

The structure of the folders and files is more or less similar to the previous version with the following exceptions:

actions.py is no longer in the root directory.
There is a new folder called actions where actions.py is located.
There is an additional file called rules.yml in the data folder.

Configuration

Version 2.0 now comes with default configurations for both the pipeline and policies. This is a lot more convenient for new users. Having said that, you can still customize it, and the format is exactly the same as the previous version.

Most of the built-in tokenizers now have been standardized with the following properties:

intent_tokenization_flag — Flag to check whether to split intents
intent_split_symbol — Symbol on which intent should be split
token_pattern — Regular expression to detect tokens

These keys are used for multi-intent classification. It only works with DIETClassifier at the moment.

Have a look at the following code snippet for WhiteSpaceTokenizer:

pipeline:
- name: "WhitespaceTokenizer"
  "intent_tokenization_flag": False
  "intent_split_symbol": "_"
  "token_pattern": None

Besides, the property case_sensitive has been moved from tokenizers to featurizers. You can safely ignore this property unless you are using the following components:

KeywordIntentClassifier
SpacyNLP
RegexFeaturizer

Policies

There is an addition of a new policy called RulePolicy. It’s useful for conversations that always have a fixed behaviour. It’s slightly different from stories and should be used together with stories for better performance. You need to have RulePolicy in your config.yml in order to use Rules and Forms. It is extremely useful when implementing the following features:

one-turn interactions, such as FAQs
fallback behaviour
handling of unwanted behaviour in forms

Moreover, the following policies have been deprecated in favor of RulePolicy. You can still implement the same functionalities using RulePolicy and their respective classifiers.

Mapping policy
Fallback policy
Two-stage fallback policy
Form policy

Importers

By default, Rasa uses the RasaFileImporter. You can now create your own data parser to load training data in other formats. This is useful if your training data is from different resources.

In addition, there is an experimental feature called MultiProjectImporter which allows you to combine the dataset from multiple Rasa projects into one. For example, you can modularize your project into the following sub-project:

restaurant bot
room booking bot
chitchat bot

and combine them later on to create a full-fledged chatbot.

Domain

The data structure of domain.yml remains the same except that you need to specify the version at the top of it. You need to specify this for all of your training data files. If it is omitted, Rasa will read it as version 2.0 by default.

version: "2.0"

intents:
  - greet
  - goodbye
  - affirm
  - deny
  - mood_great
  - mood_unhappy
  - bot_challenge

NLU

For NLU training data, the new structure for version 2.0 is as follows:

version: "2.0"nlu:
- intent: greet
  examples: |
    - Hey
    - Hi
    - hello- intent: goodbye
  examples: |
    - Goodbye
    - bye bye

Metadata

In fact, you can now declare additional metadata which contains arbitrary key-value pairs. metadata is accessible by your custom components. For example, you can declare it as follows:

nlu:
- intent: greet
  metadata:
    sentiment: neutral
  examples:
  - text: |
      hi
  - text: |
      hello

In the example given above, metadata is declared at the intent level. As a result, all of the examples contain the metadata. You can declare the metadata individually for each example.

nlu:
- intent: greet
  examples:
  - text: |
      hi
    metadata:
      sentiment: neutral
  - text: |
      hello

Retrieval Intent

In the old version, retrieval intent is an experimental feature in which an intent can be categorized into smaller sub-intents. This helps a lot when building for small talk as it reduces overhead in your stories. You need to use the / symbol to separate the main intent and its sub-intent. Have a look at the following example for a chitchat intent with two sub-intents:

nlu:
- intent: chitchat/ask_name
  examples: |
    - What is your name?
    - May I know your name?- intent: chitchat/ask_weather
  examples: |
    - What is the weather?
    - May I know the current weather outside?

Unlike the normal intent, the answers have to be placed inside responses.yml instead of domain.yml. The file should be located under the data folder, and the format is exactly the same as domain.yml. This means that you can have multiple variations and specify different payloads.

responses:
  utter_chitchat/ask_name:
    - text: "I don't have a name!"
  utter_chitchat/ask_weather:
    - text: "It is sunny"
    - text: "It is raining heavily outside"

Entities

The format for entities is still the same as the previous version, together with the experimental role and group labels. If you are not aware of it, role and group labels can be used to distinguish certain concepts in the same entity. Consider the following example:

Book me a flight from [Malaysia]{"entity": "country"} to [Singapore]{"entity": "country"}.

From a human perspective, even though both of the entities refer to country, we know that the first country refers to departure while the second entity refers to destination. We can label it using this experimental feature as follows:

Book me a flight from [Malaysia]{"entity": "country", "role": "departure"} to [Singapore]{"entity": "country", "role": "destination"}.

Stories

You can combine the stories together with NLU as a single file, but it is highly recommended to separate them. The new format is a lot more verbose compared to the old version, but it does help to distinguish between intents and actions. As a result, you are less likely to make unnecessary mistakes when building your stories.

version: "2.0"stories: - story: happy path
   steps:
   - intent: greet
   - action: utter_greet
   - intent: goodbye
   - action: utter_bye

Similar to NLU, you can define metadata inside stories to store relevant information related to the story. metadata is not used in training and will not impact the performance of your stories.

Forms

Forms are now part of the training data instead of Rasa SDK. Forms require RulePolicy, which is already inside the configuration by default. First, you should define it as follows:

forms:
  your_form:
    age:
    - type: from_entity
      entity: age

Then you can specify action and active_loop fields in which Rasa will loop over and call either utter_ask_{form_name}_{slot_name} or utter_ask_{slot_name} until the entity age is filled. You need to define them in your responses.

stories:
 - story: form asking for age
   steps:
   - intent: intent_ask_age
   - action: your_form
   - active_loop: your_form

Rules

Rules describe a small part of a conversation that always has a fixed path. You can think of it as one-turn interactions in which the same answer will be returned. Unlike stories, rules do not generalize and mostly serve to answer FAQs. It is akin to the usage of trigger via MappingPolicy in domain.yml for Rasa 1.0. Let’s say the bot should always respond with utter_greet whenever users greet it. You can easily define it using rules as follows:

rules: - rule: Say `hello` whenever users greet the bot
   steps:
   - intent: greet
   - action: utter_greet

Thanks for reading this post!

Forem: Ng Wai Foong

How to Load Test Your Kafka Producers and Consumers using k6

xk6-kafka

Building k6 with the kafka extension

Running Kafka

k6 test

Import

Initialization

Default function

Teardown

Run the test

Scale the load

Extend the test

Kafka metrics in k6

Reader

Writer

More examples

Conclusion

Benchmarking Redis with k6

redis-benchmark

Options

Examples

Latency

xk6-redis

Building k6 with the redis extension

k6 Script

Running the test

Scale the load

Latency

Conclusion

Load Testing SQL Databases with k6

Build

Install the C Compiler for SQLite3

Install the Golang Toolchain

Build the k6 binary including the SQL extension

k6 Script

Import SQL module

Connect to database

Setup and Teardown the database

Query data from database

Running the test

Scale the load

About k6 Extensions

What’s New in Rasa 2.0 - Build Your Own Chatbot

Introduction

Folder and Files Hierarchy

Configuration

Policies

Importers

Domain

NLU

Metadata

Retrieval Intent

Entities

Stories

Forms

Rules

References