Forem: michael20003

Testing Kubernetes Applications with Pytest and Testkube: A Complete Guide

michael20003 — Mon, 08 Jul 2024 14:21:12 +0000

Testing modern distributed applications within Kubernetes environments can be daunting due to the complexity and the need for scalable solutions. Traditional testing tools often fall short when it comes to efficiency and agility.

However, with the advent of Kubernetes native solutions like Testkube, it's easier than ever to integrate powerful testing frameworks such as Pytest into your testing workflows. In this comprehensive guide, we'll explore how to leverage Testkube with Pytest to streamline your testing processes in Kubernetes.

Pytest - An Overview

Python remains a top choice for programming among developers due to its simplicity and robust ecosystem. Pytest, a popular framework within this ecosystem, excels in testing Python-based applications, but not only. It is preferred for its minimalistic design, flexibility, and rich feature set, which makes it ideal to test any type of application. It includes:

Minimal Design: Pytest reduces boilerplate code, facilitating quick and easy test case creation.
Flexibility: Its modular nature and extensive plugin ecosystem allow for significant customization.
Rich Feature Set: Features like auto-discovery, assertive properties, and module fixtures streamline test management.
Diverse Testing: Pytest supports both unit and integration tests, ensuring you are covered in most testing scenarios.

While Pytest is robust for testing your applications, integrating it into Kubernetes can pose challenges such as scaling and parallel execution. This is where Testkube comes into play.

Why Use Testkube To Run Pytests in Kubernetes?

Testkube is specifically designed to orchestrate and scale Pytest executions within Kubernetes, taking full advantage of Kubernetes' scalability, flexibility, and efficiency. Here's why it stands out:

Kubernetes Native: By storing tests as Kubernetes Custom Resource Definitions (CRDs), Testkube ensures compatibility and scalability.
Integration with CI/CD Tools: Testkube seamlessly integrates with existing CI/CD pipelines, enhancing end-to-end testing capabilities.
Simplified Workflow Creation: Without the need for complex scripting, Testkube facilitates the creation of detailed test workflows, allowing for better control and customization of test executions.

Creating a Pytest Test Workflow

We've created a custom Pytest image for this example, but you can also create your own. For all the files and examples shown on this blogpost, refer to this Pytest folder.

To demonstrate the power of Testkube with Pytest, let's create a simple test workflow. We first create a Pytest test that checks an API endpoint for the correct number of objects returned.

import pytest
import requests

def test_validate_object_count():
    # Send a GET request to the API endpoint
    response = requests.get("https://api.restful-api.dev/objects")

    # Assert that the response status code is 200 (OK)
    assert response.status_code == 200

    # Parse the JSON response
    data = response.json()

    # Validate the number of objects in the response
    assert len(data) == 13, f"Expected 13 objects, but received {len(data)}"

Below are the steps to set up a Test Workflow in Testkube:

Prepare Your Kubernetes Cluster: Ensure your cluster has the Testkube agent installed and configured.
Navigate to Test Workflows: In the Testkube dashboard, click on "Add A New Test Workflow" and select "Create From Scratch".
Workflow Configuration: Follow the wizard to set up your workflow. Provide the test's image, the shell command (pytest test-api-endpoint.py), and the Git repository details.

On the next screen, you have to define the source of your test.

Choose Git from the drop-down.
Provide the path to the Git repository that has the test.
Provide a path if the test file isn't in the root folder.
Check the "Use path as working directory" checkbox.

The wizard's last page shows you the final yaml spec generated based on the values you provided.

Below is the yaml spec for the Pytest workflow:

kind: TestWorkflow
apiVersion: testworkflows.testkube.io/v1
metadata:
  name: pytest
  namespace: testkube-agent
spec:
  content:
    git:
      uri: https://github.com/kubeshop/testkube-examples.git
      revision: main
      paths:
      - Pytest-Test-Workflow
  container:
    workingDir: /data/repo/Pytest-Test-Workflow
    image: atulinfracloud/pytest-executor:latest
  steps:
  - name: Run test
    shell: pytest test-api-endpoint.py

Click on "Create" to create and save the Test Workflow.

Executing a Test Workflow

Click on "Run Now" to run the workflow. Clicking on the respective execution will show you the logs, artifacts, and the underlying workflow file.

Creating a Test Workflow in Testkube is straightforward and simple. We saw that just from one yaml file, we can manage everything related to our test - code, environments, resources, and artifacts. This makes your testing process and workflows more efficient and robust.

Summary

Pytest is one of Python's most popular testing frameworks, and Testkube is the only native Kubernetes test orchestration framework. Leveraging both these tools together streamlines your testing process for Kubernetes applications.

As we saw in this post, developers can benefit from Testkube's Kubernetes capabilities and Pytest's flexibility in creating efficient Test Workflows. You can also bring in any other testing tool and create a Test Workflow, not just Pytest.

If you already have a testing tool and want to experience Test Workflows, visit our website to learn more about Testkube's capabilities and how it can transform your testing workflow. You can also join our Slack channel for any assistance.

API Testing Using RestAssured And Testkube

michael20003 — Mon, 08 Jul 2024 14:10:15 +0000

Modern applications adopt API-first design, which provides seamless communication between services and clients. This approach is becoming increasingly popular, where APIs are designed and developed before the implementation of the actual services. This ensures that APIs are treated as first-class citizens, fostering consistency, reusability, and scalability.

Ensuring the reliability and availability of these APIs is crucial, as they are the backbone of critical business processes. API testing helps validate these services and ensure they work as expected and meet the set standards.

Considering the Java ecosystem, one popular tool for testing APIs is RestAssured. Tailored for the Java ecosystem, RestAssure is a popular tool for API testing. However, managing and executing these tests is complicated as we move towards containerized applications.

In this post, we'll examine these challenges and learn how to use RestAssured with Testkube for API testing.

RestAssured

RestAssured is a Java tool for testing APIs using Java libraries. It integrates well with build tools like Maven and Gradle, making it easy to work with. It enables automation testing of APIs by allowing developers to send simple HTTP requests and focus solely on testing the API without worrying about anything else.

Below are some noteworthy features of RestAssured:

RestAssured integrates easily with other testing frameworks, such as JUnit and TestNG, making it easier to include API tests along with other automated tests.
It supports all HTTP methods, enabling you to perform complete testing of CRUD-based functionalities.
It has out-of-the-box support for XML and JSON. This means you can use features like XMLPath and JSONPath to extract variables from responses.
There is a robust set of assertion methods to validate the responses.

You can read more about RestAssured here.

Challenges using RestAssured in Kubernetes

RestAssured is a powerful tool for testing APIs in CI pipelines and local environments. However, running these tests in a Kubernetes environment has different challenges.

Setting up environments for tests with required dependencies and configurations can be complex.
Managing and scaling tests based on the load in the Kubernetes cluster can be challenging.
Avoiding interference, maintaining consistency, and running tests in isolation is problematic.
Integrating with CI/CD tools and maintaining test artifacts and logs can be a painful process.

Let us see how Testkube helps overcome these challenges.

Using RestAssured With Testkube

Testkube is a Kubernetes-native testing framework that allows you to create testing workflows in a declarative, version-controlled way. It integrates with most of the CI/CD tools and helps you automate the initiation, execution, and validation of tests as part of automated workflows.

Testkube allows you to plug in any testing tool and leverage the power of Kubernetes. It converts your tests, test suites, and other artifacts into Kubernetes CRDs, allowing you to manage them declaratively.

With Testkube, you can create Test Workflows that include everything from provisioning necessary infrastructure components to integrating seamlessly with other testing tools and orchestrating complex tests. Refer to our Test Workflows documentation to learn more.

Let's examine how to use RestAssured with Testkube for API testing. We'll create a Test workflow using Gradle and integrate RestAssure tests into it. This repo contains all the required files for this example.

Pre-requisites

Get a Testkube account.
Kubernetes cluster - we're using a local Minikube cluster.
Testkube Agent configured on the cluster.

Once the prerequisites are in place, you should have a target Kubernetes cluster ready with a Testkube agent configured.

Creating a Test Workflow

Navigate to the Test Workflows tab and click on "Add a new test workflow"

This will provide you with three options:

Create from scratch - use the wizard to create a Test Workflow.
Start from an example - use existing k6, cypress, and playwright examples
Import from yaml - import your own Test Workflow.

We'll choose the "create from scratch" option to create this workflow.

Provide a name for the workflow and choose the type as Gradle.
Provide the run command. In this case, we'll provide gradlew test
Provide a Gradle version, we'll use 8.5.0-jdk11

On the next screen, provide the source for the test file. This can either be a Git Repo, String or a file. In this case, we'll use a Git repo.

On the next screen, it will generate the yaml spec file and display the output.

kind: TestWorkflow
apiVersion: testworkflows.testkube.io/v1
metadata:
  name: gradle-restassured
  namespace: testkube
  labels:
    test-workflow-templates: "yes"
spec:
  use:
  - name: official--gradle--beta
    config:
      run: ./gradlew test
      version: 8.5.0-jdk11
  content:
    git:
      uri: https://github.com/kubeshop/testkube-examples.git
      revision: main
      paths:
      - RestAssured Test Using Gradle
  container:
    workingDir: /data/repo/RestAssured Test Using Gradle
  steps:
  - artifacts:
      workingDir: /data/repo/RestAssured Test Using Gradle/app/build/
      paths:
      - '**/*'

The yaml file is self-explanatory as it lists down the details you've provided in the yaml. We have added extra parameters for artifacts that will collect the reports generated by RestAssured and store them.

Below is the RestAssured test file that explains what we are testing.

package org.example;

import io.restassured.RestAssured;
import io.restassured.response.Response;
import org.junit.jupiter.api.Test;

import static io.restassured.RestAssured.*;
import static org.hamcrest.Matchers.*;

public class ApiTest {

    @Test
    public void testGetEndpoint() {
        RestAssured.baseURI = "https://jsonplaceholder.typicode.com";

        given().
            when().
            get("/posts/1").
            then().
            statusCode(200).
            body("userId", equalTo(1)).
            body("id", equalTo(1)).
            body("title", not(empty())).
            body("body", not(empty()));
    }
}

The test checks for the response from https://jsonplaceholder.typicode.com endpoint and validates if the response.

The repo contains other files, including test steps and a test runner, which contain related code for executing the RestAssured test using Gradle.

Click on "Create" to create the test workflow.

Executing the Test Workflow

Once the workflow is ready, you'll see the newly created test workflow on the screen. Click on it and click "Run Now" to start the workflow.

You'll see the workflow executing along with the real-time logs of every step.

You'll see the test result based on the test execution. In this case, you'll see the test pass.

Since we have configured the artifacts for this, you can navigate to the artifacts tab and look at the reports generated by RestAssured. Testkube saves these reports for every execution making it easier to analyze the tests.

This was a simple demo of creating a RestAssured Test Workflow using Gradle for Kubernetes testing. To take advantage of test workflows more, you can create custom workflows and import them to Testkube.

Summary

In this post we looked at how API first design approach is getting popular. Many Java applications are also adopting this design principle, and tools like RestAssured are gaining popularity in testing APIs. We also looked at RestAssured and the complexities of running it on Kubernetes.

We then saw how using RestAssured with Testkube, you can create an end-to-end workflow for API testing leveraging the power of Kubernetes.

Visit the Testkube website to learn more about the other testing tools you can integrate with. If you struggle with anything, feel free to post a note in our active Slack community.

Automated Acceptance Testing with Robot Framework and Testkube

michael20003 — Mon, 03 Jun 2024 15:23:22 +0000

Introduction

Testing an application's capability usually requires setting up a production-like environment to run tests with realistic insights into the functionality and performance of the application. Tests can be both functional and non-functional, all depending on the requirements on the application and its infrastructure.

Teams usually prefer a testing solution that can be easily integrated into the existing CI/CD infrastructure without much work. But what if your testing tool can not be easily automated with your CI/CD solution or run in your existing Kubernetes infrastructure?

We decided to explore Testkube, which can take the test as code for any testing tool, automate test execution and integrate with CI/CD, and generate artifacts in Kubernetes. This would allow us to perform testing seamlessly and leverage the benefits of Kubernetes. So let's learn in this blog about acceptance testing with the Robot Framework and how to automate it with Testkube.

What is Acceptance Testing?

Acceptance testing is a phase in software development where the software is tested to determine if it meets the requirements specified by the client or stakeholders. It's usually performed after system testing and before deployment. Acceptance tests are typically written from a user's perspective and aim to verify that the system behaves as expected in real-world scenarios.

Acceptance testing also allows you to ensure software complies with legal and regulatory requirements. This reduces the risk of legal consequences or penalties due to non-compliance, especially in certain industries, such as healthcare or finance.

Other testing practices, such as API, load, etc., contribute to different aspects of software quality, such as performance, reliability, and interoperability, and are usually done before acceptance testing.

The Robot Framework is an acceptance testing tool that is easy to write and manage due to its key-driven approach. Let us learn more about the Robot Framework to enable acceptance testing.

Acceptance Test with Robot Framework

Robot Framework is an open-source, keyword-driven test automation framework that supports Python, Java, and .NET and is widely used for acceptance testing, acceptance test-driven development (ATDD), and robotic process automation (RPA). It provides a highly readable, tabular syntax for writing test cases in a plain-text format, making it accessible to both technical and non-technical users.

Robot Framework stands out for the following reasons:

Keyword-Driven Approach: In Robot Framework, test cases are defined using keywords that represent the validation. These keywords are organized into test suites, which are further structured into test cases.
Test Data and Variables: Test cases in Robot Framework are written in a tabular format using plain-text files, typically in files with the .robot extension.
Built-in and Custom Keywords: Robot Framework offers a set of built-in keywords for performing common actions such as clicking buttons, entering text, verifying text, etc. It also allows users to define their custom keywords.
Library Support: Robot Framework supports integration with external libraries written in Python, Java, .NET, and other languages, allowing users to extend its functionality as needed. These libraries can provide additional keywords for interacting with specific technologies, such as web browsers, APIs, databases, and more.

Test cases written in Robot Framework can be executed using the robot command-line tool or through various integrated development environments (IDEs) and continuous integration (CI) systems. During execution, Robot Framework generates detailed reports and logs, highlighting the status of each test case and providing insights into test results and any encountered issues.

Running Acceptance tests in Kubernetes

While running Robot Framework tests against legacy/non-containerized applications is usually straight-forward, there are several challenges associated with running these tests against a containerized application infrastructure built on Kubernetes:

Resource Management: Kubernetes manages containerized applications, including their resource allocation and scaling. However, scaling requires ensuring optimal resource utilization for Robot Framework test execution. This would be challenging to deal with, especially in case of varying test workloads and resource requirements.
Test Isolation: Ensuring proper isolation for individual test executions can be complex while running multiple tests concurrently. It will require careful consideration to avoid resource conflict and interference between tests.
Persistent Data Handling: Robot Framework may require access to persistent data, such as test artifacts, test data files, or external dependencies. Managing persistent data in a Kubernetes environment, including storage provisioning, volume mounting, and data synchronization, can be challenging, especially when dealing with distributed test executions.
Secure Access: Access to applications or components running in Kubernetes can be severely constrained for security reasons, making it difficult to run tests against them with tools that are not running in the cluster themselves. Relaxing network policies for the sake of testing is not always an option.

Thus, when using the Robot Framework for acceptance testing of your containerized applications, you will require a test-execution framework that eases the life of a developer and can orchestrate Robot Framework tests with all the above caveats in mind. Fortunately,there is such a framework specifically targeted at running any kind of testing tool, including the Robot Framework, in a containerized application infrastructure: Testkube.

Acceptance Testing with Testkube

Testkube is a test orchestration platform that supports the execution of any testing tool in a containerized application infrastructure. It leverages Kubernetes to provide a scalable and consistent execution environment for all your tests, and includes a unified dashboard for test definition, reporting, and troubleshooting.

To run tests in Testkube, one defines Test Workflows for managing the execution lifecycle of individual test(s), which will then be executed in your Kubernetes environment using the corresponding testing tool image. Let us understand more about the Test Workflows feature.

Test Workflows

Test Workflows provide a declarative approach for defining test execution lifecycles. It leverages Kubernetes-native capabilities to orchestrate and execute tests, collect real-time artifacts, and display them on the Testkube dashboard. A Test Workflow is defined as a single YAML configuration that is stored as a custom resource in your Kubernetes cluster(s), Test Workflows allow integration with existing testing tools and GitOps pipelines, and also provide templates for standardized testing configurations adaptable to various needs.

Execute Acceptance Testing in Testkube

To run a Robot Framework acceptance test with Testkube, you will need to define a corresponding TestWorkflow that uses a prepackaged Robot container image to run the test cases that you have defined - let's have a look at how that works.

We will see two demos here using the Robot Framework in Testkube using the sample example, Restful Booker, provided by Robot Framework.

Basic: In the basic execution we will run the sample test case by Restful Booker. In this test case, we will get bookings for a user, create a booking for a user, and verify that the application performs the required action. All these user details are hardcoded in the test case. We will execute the test in the Testkube dashboard and collect artifacts.
Advanced: We will update the above test case to use variables instead of hard coding user details and make use of parameters to provide values. This will give us the flexibility to test different scenarios. In Testkube we will make use of parameters to process these parameterized inputs.

So let us see Testkube in action performing an acceptance test using Robot Framework. Before we get started, let us verify the following prerequisites.

Pre-requisites

To follow along, you will need to set up the following:

A Testkube Pro account (free plan is fine)
A Kubernetes cluster - we're using a local Minikube cluster (https://minikube.sigs.k8s.io)
The Testkube Agent is configured on the cluster.

Once the prerequisites are in place, you should have a target Kubernetes cluster ready with a Testkube agent configured.

Basic Execution

In the basic execution, we will run the example Restful Booker test code in Testkube using Test Workflows. We have pushed the test case to a GitHub repository. These files will by default be mounted by Testkube to the '/data/repo' directory when the test executes. Let us get started.

Select Test Workflows from the menu and click on the 'Add a new test workflow' button to start setting up the workflow.
From the options to 'Create a Test Workflow', select 'Create from scratch'.
Enter the following details to create a new workflow.
a. Name: Set the name of the workflow as 'basic-acceptance-test'
b. Type: Select the type of test as 'Other'
c. Image: Enter the name of the Docker Container Image for Robot Framework. We are using 'ppodgorsek/robot-framework'
d. Tag: Search for the stable tag on Docker Hub for the image and enter it here. We have been using 7.1.0 since it was recently released.
e. Shell command: By default, the repository that has the test case is mounted to the /data/repo directory. We will use the robot command line tool to execute the test, and using the –outputdir option, we will save the output to a specific location. Here is the shell command: 'robot --outputdir /data/repo/output /data/repo/test-restful.robot'.
Create a test workflow with your custom details.
Select the Source as Git from the drop-down menu and enter the repository details.
In the final step, the UI will create the YAML for our Test Workflow. Let us go ahead and add the path where the artifacts will be stored in spec.steps. Here is what the final configuration looks like.

kind: TestWorkflow
apiVersion: testworkflows.testkube.io/v1
metadata:
 name: basic-acceptance-test
spec:
 content:
   git:
     uri: https://github.com/cerebro1/robot-framework-test.git
 container:
   image: ppodgorsek/robot-framework:7.1.0
 steps:
   - name: Run test
     shell: robot --outputdir /data/repo/output /data/repo/test-restful.robot
 artifacts:
   paths:
     - /data/repo/output/*

Click on Create and your 'basic-acceptance-test' workflow is ready:
Test Workflow created in Testkube
Click on 'Run this Test Workflow now' to execute the workflow. Once the Test Workflow executes successfully, you will see the Test Workflow in the 'Recent executions' list.
Test Workflow execution is complete and successful.
Select the Test Workflow to view the details. You can see each step has been executed successfully in the following image. This is helpful for debugging in case something goes wrong or you need more insights. Test Workflow has a separate section for Log Output, Artifacts, and Workflow.
Test Workflow Log Output shows each step execution.
Click on 'Run shell command' to view the details of the test cases executed. The execution shows that all the tests have passed successfully.
Robot Framework test cases executed in Test Workflow
Click on the Artifacts tab to view the artifacts processed. You can download the artifacts generated by the Robot Framework here.
Test Workflow successfully processes the artifacts.

Thus, with the help of Test Workflow, we were able to run a Robot Framework test seamlessly without any setup required on the local machine. The output shows all the tests have passed. Testkube has processed the artifacts and provided them for download.

Now, let us suppose, this Robot Framework test has some parameterized values to be passed with the test. Don't worry! Testkube has got you covered. In the Test Workflows, you can pass the parameters and provide custom values. Let us understand this in detail in the coming section.

Advanced Execution with Parameters

Variables in Robot Framework

In Robot Framework, you can define the parameters as variables, and the values can be passed at the time of execution. The default values to these parameters can be set in the Robot Framework test configuration as shown below:

*** Variables ***
${my_var} my_default_value

*** Test Cases ***
Test Case 1
    Log ${my_var}

While using the robot command line utility, these variables can be passed as parameters as shown below:

$ robot --variable my_var:my_value test.robot

Robot Framework Variables in Test Workflows

Test Workflows allows you to define configuration parameters for your workflows, which can be passed to the Robot Framework as input. Let's add parameters for the firstname, lastname and totalprice parameters to our workflow:

kind: TestWorkflow
apiVersion: testworkflows.testkube.io/v1
metadata:
  name: basic-acceptance-test
  namespace: testkube
spec:
  config:
    firstname:
      type: string
      default: Jim
    lastname:
      type: string  
      default: Henderson
    totalprice:
      type: string
      default: "100"
  content:
    git: 
      uri: https://github.com/cerebro1/robot-framework-test.git
  container:
    image: ppodgorsek/robot-framework:7.1.0
  steps:
    - name: Run test
      shell: robot --variable firstname:{{ config.firstname }} --variable lastname:{{config.lastname }} --variable totalprice:{{ config.totalprice }} --outputdir /data/repo/output-{{config.firstname}}-{{config.lastname}} /data/repo/advanced-test-restful.robot    
  artifacts:
    paths:
      - /data/repo/output-{{config.firstname}}-{{config.lastname}}/*

As you can see we have defined three config variables which we then pass to the robot command, and use in the naming of the output folder. When we new try to run this test we are first prompted to provide values for our variables:

Just pressing the run button uses the default values provided as you can see in the log output:

Integrating Testkube with CI/CD

Although it is useful to run tests directly from the Testkube Dashboard as shown above, you will most likely want to start these tests from your existing CI/CD pipelines. Fortunately, Testkube makes this really easy:

Navigate to the CI/CD Integration tab for your Test Workflow (see below)
Select the CI/CD tool you are using from the menu to the left - a corresponding example will be shown to the right that you can copy/paste directly into your tool

Conclusion

Using Test Workflows, we were able to easily configure the execution of our Robot Framework test, and also pass custom variables to Robot Framework in a seamless way, and automate the execution of our tests from our CI/CD system. The configuration is easy to understand and manage, which not only reduces the efforts of developers but also increases the maintainability of the configuration.

For a scalable approach to test execution in a Kubernetes environment, Testkube is the solution. It takes any testing tool, provides a template, or allows you to start from scratch and automate your tests. Teams can manage the entire test life cycle in an optimized way from a single dashboard. This not only empowers them with the feasibility of scaling but also standardizes the testing process.

In this tutorial, we have seen how acceptance testing with Robot Framework could be done using Test Workflows in Testkube. Similarly, teams can create multiple test workflows to manage unit, load, API, acceptance, etc., tests from the same platform.

We invite you to try Test Workflows. Visit Testkube website to get started. If you find yourself struggling with anything, feel free to drop a note in our active Slack community, and someone will help you out.

Stop Using Docker in Docker for Testing

michael20003 — Mon, 03 Jun 2024 13:17:53 +0000

Embrace Kubernetes to Improve Performance, Scalability, and Ease of Configuration for your Tests

Testing applications is becoming increasingly difficult as they grow in size and complexity. For someone responsible for testing an application of that scale, simply running tests within individual containers doesn't help much. Hence, there is a need for a scalable and flexible approach that allows for more efficient and effective testing to catch those pesky bugs.

One technique that gained popularity for testing distributed applications was using Docker-in-Docker or DinD. The idea was to run a Docker daemon inside a Docker container itself and then use that inner Docker environment to orchestrate the full application stack on a single machine.

Initially, it provided a seemingly straightforward way to create realistic multi-container environments for integration testing. However, as the applications grew, teams realized their substantial drawbacks and limitations, from resource constraints and overheads to inconsistencies.

If you're responsible for testing in your organization, you can relate to the above scenario very well. Testing in Docker in Docker comes with its issues, and for large applications, it can become a nightmare. In this blog post, we'll look at the challenges with Docker in Docker approach and why you should embrace Kubernetes for Testing.

Docker in Docker 101 (DinD)

But before we further delve into this blog post, a short primer on Docker in Docker (DinD) is necessary. As the name suggests, it's running a docker container inside a docker container. The approach first surfaced in late 2013, and it soon gained popularity.

One common use case of DinD is continuous integration (CI) pipelines. The CI system could start up a DinD container as part of the build process. This inner Docker is then used to pull application images, start their containers, and run tests within that isolated environment.

While DinD can be convenient for certain workflows as mentioned above, it introduces a range of complexities and limitations. Let us discuss them in the following section.

Problems with Running Docker In Docker

Docker-in-Docker seemed like a convenient solution for testing distributed applications initially, but the reality is that this approach has significant drawbacks that become more evident as your applications grow in complexity. From a testing standpoint, DinD introduces several challenges that can undermine the very goals you're trying to achieve.

Inefficient Performance

One of the major drawbacks of the DinD approach is its sub-optimal or inefficient performance, especially at scale. The DinD testing approach hogs your resources if not managed correctly. Because each docker daemon reserves a set of resources, including CPU, RAM, networking, etc, on a system with limited resources, this overhead can quickly lead to performance bottlenecks and instability while testing.

Inability to Mirror Production Setup

Testing an application in a production-like environment is vital. However, the DinD environment fails to accurately mimic your actual production setup. The nested container environment may exhibit different behavior due to different storage, networking, and security configurations that may not be similar to your production environment. For instance, your production containers may run on a different Docker runtime (containerd, crio etc) compared to what DinD provides by default.

Docker Security Concerns

When you have applications dealing with sensitive data, you need to be extremely cautious when testing with the DinD approach, as it can open up new vulnerabilities and security risks if not configured properly. By default, the Docker daemon runs with root privileges, if not configured correctly, it could expose your application data or increase security loopholes. Furthermore, concerning multi-cluster/tenant setup, there are security concerns of isolation between the different nested Docker instances, potentially allowing cross-tenant data leaks and vulnerabilities.

Docker Scalability Concerns

With DinD, you end up with limited resources available on a single host machine. Even if multiple DinD instances are utilized, efficiently utilizing resources and automating scaling becomes a difficult process without an orchestrator. The lack of out-of-the-box scaling options severely limits the ability to conduct rigorous integration, load, and performance tests. Teams struggle to accurately model real-world scale, resiliency scenarios, and sophisticated deployment requirements.

From performance overheads to security and scaling concerns, the technical debt of using DinD stacks up rapidly, and continued usage of this approach may lead to maintenance overhead. However, the major concern is that DinD severely restricts your ability to conduct comprehensive integration, load, and other performance tests on a large scale.

Even the creator of DinD asks you to think twice before using DinD for your CI or testing environment.

Why Embrace Kubernetes For Testing?

As testing needs evolve, the Docker in Docker testing approach fails to provide the scalability, flexibility, and resilience that modern distributed systems demand. This realization led us to explore Kubernetes as a more robust solution for modeling and testing our cloud-native applications. Let us look at how embracing Kubernetes for Testing overcomes the issues with DinD.

Built-in Scalability

Kubernetes comes with built-in orchestration capabilities that enable on-demand scaling, which is impossible with the DinD approach. It provides mechanisms like horizontal pod autoscaling, cluster autoscaling, and the ability to automatically schedule containers across a pool of nodes. This allows you to model realistic load scenarios by scaling application components using simple commands or auto-scaling rules based on CPU/memory consumption.

Consistent Production-Mirrored Environments

The core goal of any testing process is to ensure testing takes place in an environment that closely resembles the production environment. Today, Kubernetes is dubbed the OS of the cloud and has become a standard for deploying and managing applications on the cloud. This allows your test clusters to accurately mirror your production environment, leading to effective testing.

Simplified and Efficient

Kubernetes is known to be a complex tool to master and involves a steep learning curve at the start. However, as you progress through the Kubernetes journey, things become simpler. The rise of cloud-native CI/CD solutions built specifically for Kubernetes has further contributed to standardizing testing workflows. Instead of building custom pipelines, teams can leverage pre-built integrations for spinning up test clusters, running test suites within live environments, collecting telemetry data, and more.

These were just a few reasons why we should embrace Kubernetes for testing. It brings in a lot of advantages that make your testing processes more efficient. However, one of the challenges when it comes to testing in Kubernetes is the lack of Kubernetes native testing tools and frameworks, which limits your ability to take full advantage of Kubernetes. This is where Testkube comes in.

Run Your Tests in Kubernetes With Testkube

Being the only Kubernetes native test execution and orchestration framework, Testkube takes your testing efforts to the next level. It helps you to build and optimize your test workflows for Kubernetes. It has a suite of advantages, some of which are listed below:

You can plug in any testing tool to Testkube, make it Kubernetes-native and take full advantage of Kubernetes. Check the complete list of Testkube integrations.
With Testkube you can easily create and manage complex test scenarios across multiple environments.
Testkube integrates with a suite of CI/CD tools to enhance your DevOps pipelines by enabling complete end-to-end testing of your applications.

Moving your testing framework from DinD to Kubernetes might seem intimidating, but with Testkube, it's a breeze. As mentioned earlier, Testkube offers a suite of features along with tools and integrations that streamline test deployment, execution and monitoring within Kubernetes ensuring your tests are as close to the production environment as possible.

Using Testkube Test Workflows

Testkube allows you to create end-to-end test workflows using your favorite testing tool. It also streamlines your testing process by decoupling test execution from your CI pipeline. This democratizes the testing process, allowing teams to execute tests on-demand without disrupting the CI workflow. By separating test execution from the CI pipeline, Testkube offers greater flexibility and control over the testing process.

In this section, we'll examine how to create a test workflow using Testkube. We'll create one for Postman, but you can configure a workflow for virtually any testing tool.

Creating A Test Workflow Using Postman

In order to create a test workflow, you need to have a cluster running along with the Testkube agent. When you have this running, you can log in to the Testkube dashboard, click on Test Workflows and "Add A New Test Workflow" and choose "Start From An Example"

By default, a k6 test spec is provided. We'll be using Postman instead. To do so, click on the "Discover more examples" link, which will take you to the Test Workflows document. Scroll down to the Postman section and copy the code present there. You can also use the one shown below:

apiVersion: testworkflows.testkube.io/v1
kind: TestWorkflow
metadata:
  name: postman-workflow-example # name of the Test Workflow
spec:
  content:
    git: # checking out from git repository
      uri: https://github.com/kubeshop/testkube
      revision: main
      paths:
        - test/postman/executor-tests/postman-executor-smoke-without-envs.postman_collection.json
  container: # container settings
    resources: # resource settings (optional)
      requests: # resource requests
        cpu: 256m
        memory: 128Mi
    workingDir: /data/repo/test/postman/executor-tests # default workingDir
  steps: # steps that will be executed by this Test Workflow
    - name: Run test
      run:
        image: postman/newman:6-alpine # image used while running specific step
        args: # args passed to the container
          - run
          - postman-executor-smoke-without-envs.postman_collection.json

Paste this in the spec section and click Create. This will create the test workflow and take you to the workflow page.

Executing a Test Workflow

Click on "Run Now" to start the workflow execution. Clicking on the respective execution will show you the logs, artifacts, and the underlying workflow file.

Testkube Workflows preserve the logs and artifacts generated from the tests, thus providing better observability. Creating a test workflow is fairly straightforward, as shown above, and you can bring your testing tool and create a workflow for it with a few clicks.

Summary

The move from Docker in Docker to Kubernetes for testing is the need of the hour, especially considering the scale and complexity of our applications. By leveraging Kubernetes for testing, we take advantage of its inherent capabilities of scaling, security and efficiency to name a few. And with tools like Testkube, organizations can achieve more reliable, efficient, and cost-effective testing processes. If you are ready to elevate your testing strategy with Kubernetes and Testkube feel free to visit our website to learn more about Testkube's capabilities and how it can transform your testing workflow.

Integrating Testkube With Azure Pipelines For Continuous Testing

michael20003 — Tue, 14 May 2024 16:53:47 +0000

In the fast-paced world of software development, agility is key. Teams strive to expedite feature releases, yet the critical process of testing often lags behind. To truly achieve agility and ensure reliability, automated testing must be woven throughout the development lifecycle.

Enter the concept of Continuous Testing, or TestOps, which embodies the integration of testing into DevOps practices. This isn't just about triggering tests upon code commits or automating report generation; it's about embedding a robust testing framework within your CI/CD ecosystem.

Testkube emerges as a solution specifically designed to facilitate testing in Kubernetes environments and to dovetail with CI/CD workflows effortlessly. This blog post will explore the synergy between Testkube and Azure Pipelines, illustrating how this integration enhances continuous testing practices.

Leveraging Testkube with Azure Pipelines for Enhanced TestOps

Azure Pipelines, a pivotal component of Microsoft Azure's DevOps services, offers seamless integration with leading version control systems like GitHub and GitLab. It utilizes an azure-pipelines.yaml file to manage pipeline triggers and job configurations, enabling automated builds and deployments with ease.

Testkube, on the other hand, is a versatile testing framework designed to harmonize with various CI/CD tools, bringing your preferred testing mechanisms into the Kubernetes ecosystem. By treating tests as Kubernetes resources through Custom Resource Definitions (CRDs), Testkube revolutionizes test management by allowing tests to be handled as code.

Integrating Testkube with Azure Pipelines unlocks significant advantages:

Enhanced TestOps: This integration erases the traditional boundaries between testing and development, fostering a culture of continuous testing. It ensures product reliability by facilitating testing at every stage of the development process, accommodating rapid scaling and deployment speeds.
Seamless Continuous Testing: With Testkube and Azure Pipelines, testing becomes a continuous process that automatically triggers with every code commit. This ensures comprehensive testing with each change, embedding quality directly into the CI/CD pipeline.
In-depth Insights: Testkube excels in capturing and analyzing data from your executed tests. It leverages this information to generate detailed insights and dashboards, offering a clear view of test performance and trends over time.

This integration not only streamlines workflows but also empowers teams to maintain high-quality standards, ensuring agile, reliable product development with Testkube and Azure Pipelines.

Integrating Azure Pipelines With TestKube

Integrating Testkube with Azure Pipelines follows a process similar to what we did for GitLab and CircleCI in the previous posts. In this case, we'll configure Azure Pipelines with the GitHub Repo used in an earlier blog post.

Create a DevOps organization in Azure and then create a new project to get started.
On the first run, it will ask you to configure your code repo with Azure Pipeline.
Create an azure-pipelines.yml in the root of your GitHub repository.
The pipeline will create and execute a k6 test on a cluster with Testkube configured.
Lastly, we'll configure a branch protection rule that will not allow a change to be merged if the test fails.

At a high level, this is what we are going to do. The code for this blog post is available in this GitHub repo. It has a folder named k6, which has the test, and an azure-pipelines.yaml file with the Azure Pipeline configuration.

Pre-requisites

Azure DevOps organization & repository with Azure Pipeline configured.
Kubernetes cluster - we're using an AKS cluster.
Testkube Agent configured on the cluster.
Testkube API token, Org ID, and Environment ID

Once the prerequisites are in place, you should have a target Kubernetes cluster ready with a Testkube agent configured. You should also have a GitHub repository configured with Azure Pipelines integrated with Testkube.

Note: Refer to our API Token document to create the API token. To find the Org ID and environment IDs, log in to your Teskube Cloud dashboard, head to any existing tests page, navigate to CI/CD integration, and choose GitHub. Copy the environment and organization IDs from here.

You'll also need to create variables in Azure Pipelines to store these IDs and the token.

K6 Test

We will create a k6 test that performs a load test on the testkube.io website and checks if the response time is less than 1000 milliseconds. If the response is more than that, the test will fail.

import http from 'k6/http';
import { check, sleep } from 'k6';

export let options = {
  vus: 1,            // Virtual Users
  duration: '10s',   // Duration of the test
};

export default function () {
  // Send an HTTP GET request to the website
  let response = http.get('https://testkube.io');   // Replace with your URL

  // Check if the response time is less than 1000 milliseconds
  check(response, {
    'Response time is less than 1000 milliseconds': (r) => r.timings.duration < 1000,   // Check if response time is less than 1000 milliseconds
  });

  // Add a sleep period (in this case, 1 second) between requests
  sleep(1);
}

Configuring Azure Pipeline

We configure the Azure Pipeline to execute whenever a new pull request is made. The job uses the Testkube API key, ORG_ID, and ENV_ID, saved as Azure Pipeline variables.

trigger:
  - main

pool:
  vmImage: 'ubuntu-latest'

stages:
  - stage: Test
    jobs:
      - job: RunTestkube
        steps:
          - task: SetupTestkube@1
            inputs:
              organization: '$(TK_ORG_ID)'
              environment: '$(TK_ENV_ID)'
              token: '$(TK_API_TOKEN)'
          - script: |
              git clone https://github.com/techmaharaj/testkube-GH.git
              cd testkube-GH/k6
              testkube create test --name k6-test --type k6/script --update -f testkube.js
              testkube run test k6-test
            displayName: Run Testkube Test

Pro Tip: You can also use this sample workflow generated by Testkube to create and configure your Azure Pipeline.

The above workflow does the following things:

Configures a job to connect with Testkube cloud using the Testkube token, organization, and environment ID.
Creates and executes the k6 test using kubeshop/testkube-cli on the AKS cluster.
Based on the result, the status will be updated in the pull request. If the test passes, the merge option will be enabled, and anyone can merge it. However, if the test fails, the merge will not be available - we'll configure a branch protection rule to prevent the PR from being merged if the run fails.

Configuring GitHub Repo

You first need to create an Azure DevOps organization. Once this is made, you'll be prompted to add your first project. In this case, we're using a GitHub project. So, we can choose GitHub and follow the prompts to connect your GitHub repository to Azure DevOps.

After successfully creating the project, you need to make your pipeline. Azure will provide you with a default pipeline code that you can commit to your repo. However, since we already have our configuration, we will use that.

At this point, our Azure Pipeline is configured to be triggered on every new PR on the main branch of the GitHub repository.

Trigger Testkube Test

To check if everything works correctly, raise a new pull request in the repository with some changes. You'll see that your action is triggered almost immediately.

You see two jobs here, that is because we already have a GitHub action enabled in this repository from our previous blog post.

At the same time, you can also see the execution on Azure Pipeline. In the background, our test will run on the target Kubernetes cluster specified.

If the test passes, our pipeline will update the status in the PR, and a user can merge the PR.

However, if the workflow fails, the CI job will update the workflow status as failed in the PR. Merging will be blocked because of the branch protection rule that we have created, which prevents merging if the test fails.

Conclusion: Revolutionizing Testing with Testkube and Azure Pipelines

In this blogpost, we learned about the pivotal role of continuous testing in modern software development, spotlighting how it significantly uplifts testing practices. We delved into the synergy between Testkube and Azure Pipelines, demonstrating a seamless integration that facilitates an efficient CI/CD workflow. By integrating a k6 test with Testkube and executing it through Azure Pipelines, we showcased a practical workflow that enhances testing efficiency and reliability.

For a deeper dive and more comprehensive guidance on harnessing Testkube within your CI/CD ecosystem, our documentation offers extensive insights. Embrace TestOps by experimenting with Testkube and witness firsthand the transformation it brings to your testing strategy.

Should you have any queries, need guidance, or wish to share feedback, our Slack Community is a vibrant space for support and knowledge exchange. Embark on your journey towards more proactive, streamlined testing today.

Happy testing!

Critical Test Based Alerting with PagerDuty and Testkube

michael20003 — Thu, 09 May 2024 19:12:40 +0000

In the realm of agile development, the focus often lands on accelerating feature deployment. However, the capability to swiftly respond to critical incidents is just as crucial. Especially in testing, where the failure of key tests could significantly impact a release, staying updated on test statuses with each build is vital.

Modern applications and infrastructures require effective alerting and incident response systems to maintain high availability and operational awareness. PagerDuty stands out as a leading solution for incident management, providing teams with the tools needed to quickly address and resolve issues.

This tutorial post explores how integrating PagerDuty with Testkube can enhance your team's response to critical test failures, ensuring your development cycle remains both fast and reliable.

Alerting Via Webhooks

Throughout the evolution of technology, numerous methods have enabled systems to communicate seamlessly. From a variety of protocols to services, the goal has always been to facilitate the transfer of data across disparate systems efficiently.

Webhooks stand out as a powerful mechanism for system interaction. They serve as digital informants, alerting services to specific events. Essentially, a webhook endpoint acts as a listener for one system, ready to receive data transmitted by another.

In the context of Testkube, Webhooks are instrumental. They empower you to inform other services about test-related activities effortlessly. For example, setting up a Slack webhook with Testkube allows for the automatic dissemination of event notifications, bridging the gap between test execution and team communication.

Key Components of a Webhook Trigger in Testkube:

Events: These are the specific occurrences within Testkube you wish to monitor, such as "test-start" or "test-fail". Selecting relevant events allows for targeted notifications.
Source: Indicates the origin of the event, whether it's an individual test or a collection of tests within a TestSuite.
Destination: The webhook endpoint or the digital address to which Testkube sends the notification.

Advantages of Implementing Webhooks

Instant Alerts: Stay informed with real-time notifications about your tests' outcomes, enabling immediate action.
Automated Workflows: Trigger specific actions based on test results, such as sending detailed emails after a failure, streamlining your response mechanism.
Seamless Integration: Webhooks facilitate communication between Testkube and various tools or services like Slack for messaging or Grafana for monitoring, effortlessly fitting into your existing workflows.

By utilizing webhooks within Testkube, you can significantly enhance test observability and automation, making your testing processes more efficient and integrated.

Integrating Testkube With PagerDuty

PagerDuty is an incident response management and alerting platform for DevOps teams managing critical operations. It provides real-time alerts and escalates incidents to the appropriate personnel. By automating the incident response workflow, PagerDuty ensures that incidents are addressed promptly and efficiently, reducing downtime and improving overall system reliability.

Being the extensive tool that it is, there are a few components that we must be aware of for the purpose of this tutorial:

Services: These are applications/services within an organization that are monitored. These are connected to monitoring tools that trigger alerts based on predefined rules and thresholds.
**Integrations: **Integrations allow PagerDuty to seamlessly connect with other monitoring and collaboration tools. This allows it to collect data from multiple sources and manage them centrally.
Incidents: These are events that indicate something has gone wrong. When a monitoring tool detects an issue, it sends an alert to PagerDuty, which creates an incident that can be categorized and assigned to different personnel.

Apart from these, PagerDuty has advanced mechanisms that allow teams to send emails, calls, SMS, and push notifications to notify designated personnel about the incident. You can read more about PagerDuty and its features here.

Creating a service & integration

The first and foremost step is to create a service in PagerDuty. Creating a service in PagerDuty is fairly simple, and you can create one in a few minutes.

After your service is created, you need to create an integration. And integration basically allows PagerDuty to receive alerts from other services/tools. By default, PagerDuty provides support from multiple tools. However, in the case of Testkube, we’ll use their Events API V2, which will allow us to send custom events.

After you select this, it will give you the configurations that you will use to create the webhook in Testkube. It will share details like integration key, integration URL for change and alert events along with an example to use the API. Note that PagerDuty requires the input to come in a specific format.

With this, we have configured PagerDuty to receive alerts from Teskube. Let us now go ahead and create a Webhook in Testkube.

Creating a Testkube Webhook

Creating a webhook trigger in Testkube is straightforward. You can create Webhooks using the dashboard as well as the CLI. We’ll be using the dashboard to show you how it works.

Using Dashboard

We’ll set up a webhook for PagerDuty using the dashboard. This webhook will trigger on the test-end-failure event and send the data to the PagerDuty webhook endpoint that we got in the previous step (Integration URL)

Navigate to the webhooks section and “Create a new webhook.”

Provide a name for your webhook.
Choose the “Resource Identifier” as any test or executor on which you want the webhook to trigger. In this case, we will use a simple curl test. You can find the test in this repo - we’ve changed the HTTP code in the test so that it fails.
For “Triggered events”, choose end-test-failed, which basically means it will trigger when the chosen Pytest fails.

Click on Submit, and your webhook is ready. However, as mentioned earlier, PagerDuty requires input in a particular format as shown below:

curl --request 'POST' \
--url 'https://events.pagerduty.com/v2/enqueue' \
--header 'Content-Type: application/json' \
--data '{
  "payload": {
    "summary": "Test alert",
    "severity": "critical",
    "source": "Alert source",
    “custom_details” {“key”:”value”}
  },
  "routing_key": "routingkey",
  "event_action": "trigger"
}'

Hence, we will add a custom payload. To do that, click on your Webhook and navigate to the Action section. Here, add the following snippet under the “Custom Payload” section

{
"payload": {
    "summary": "{{ .TestExecution.TestName }} - {{ .TestExecution.ExecutionResult.Status }}" ,
    "severity": "critical",
    "source": "Testkube",
    "custom_details" : {
        "testName": "{{ .TestExecution.TestName }}",
        "testType": "{{ .TestExecution.TestType }}",
        "testStatus": "{{ .TestExecution.ExecutionResult.Status }}",
        "testEndtime": "{{ .TestExecution.EndTime }}",
        "testDuration": "{{ .TestExecution.Duration }}",
        "url": "https://app.testkube.io/organization/{{ index .Envs "TESTKUBE_CLOUD_ORG_ID" }}/environment/{{ index .Envs "TESTKUBE_CLOUD_ENV_ID" }}/dashboard/tests/{{ .TestExecution.TestName }}/executions/{{ .TestExecution.Id }}"
    }
  },
  "routing_key": "routingkey",
  "event_action": "trigger"
}

In the above payload, we are sending the following details:

Test Name
Test Type
Test Status
Test Endtime
Test Duration
URL to the specific test log

Click on save to save the configuration.

Testing the Webhook

Now that we have configured the webhook for PagerDuty, it’s time to test it. To do that, simply run the curl test.
You’ll see the webhook trigger when the test fails and send data to PagerDuty. Within PagerDuty, this will be shown as an incident based on the details we shared in the custom_payload.

By default, PagerDuty also sets up an email alert triggered when an incident is created.

And that’s how you can configure the Testkube webhook to send alerts to PagerDuty. Based on your team’s specific needs, you can configure both Webhook and Testkube.

Conclusion: Elevate Your Incident Response with Testkube and PagerDuty Integration

Harnessing the synergy between Testkube and PagerDuty through webhooks offers a formidable solution to bolster your incident management strategy. This integration not only simplifies tracking and addressing critical test failures but also fosters improved teamwork.

By utilizing Testkube's automated testing capabilities alongside PagerDuty's robust incident management system, you equip your team with the necessary resources to uphold service excellence and enhance customer satisfaction.

Discover the potential of Testkube and its versatile integrations. Embrace a system where proactive monitoring and seamless alerting transform your operational dynamics.

Connect with our vibrant community on Slack for insights, support, and networking with like-minded developers. Embark on a journey to redefine your approach to quality assurance and incident response.

Integrating Testkube and Zapier For Instant Email Alerts

michael20003 — Tue, 30 Apr 2024 18:11:27 +0000

Gone are the days of simple, standalone apps. Today's applications are complex, distributed systems spanning multiple environments and sometimes even different geographies. Ensuring every component works seamlessly is more crucial than ever, highlighting the indispensable role of comprehensive testing.

However, managing and monitoring the myriad of tests across various modules and environments poses a significant challenge. Enter Testkube, a tool not only for centralizing test metrics for enhanced observability but also for integrating these metrics into your existing systems for real-time insights.

This tutorial will guide you through setting up a Zapier webhook in Testkube to funnel test data directly to Zapier, allowing you to automatically generate and send detailed email alerts via Gmail. Stay tuned for a streamlined approach to keeping tabs on your tests.

Webhooks in Testkube

Think of a doorbell: it alerts you to someone's presence without needing to watch the door constantly. Webhooks serve a similar purpose in the digital realm, acting as notifications that prompt action or relay information between different applications when specific events occur.

Testkube leverages webhooks to inform you about test activities. Whether a test succeeds, fails, or triggers another significant event, Testkube can notify other systems in real-time, enhancing your testing workflow's responsiveness and integration.

Key Components of a Webhook Trigger in Testkube:

Events: These are the specific occurrences within Testkube you wish to monitor, such as "test-start" or "test-fail". Selecting relevant events allows for targeted notifications.
Source: Indicates the origin of the event, whether it's an individual test or a collection of tests within a TestSuite.
Destination: The webhook endpoint or the digital address to which Testkube sends the notification.

Advantages of Implementing Webhooks

Instant Alerts: Stay informed with real-time notifications about your tests' outcomes, enabling immediate action.
Automated Workflows: Trigger specific actions based on test results, such as sending detailed emails after a failure, streamlining your response mechanism.
Seamless Integration: Webhooks facilitate communication between Testkube and various tools or services like Slack for messaging or Grafana for monitoring, effortlessly fitting into your existing workflows.

By utilizing webhooks within Testkube, you can significantly enhance test observability and automation, making your testing processes more efficient and integrated.

Integrating Testkube With Zapier

Zapier is a tool that automates tasks in apps like Gmail and Slack, serving as a connector to streamline workflows without coding. At its core, Zapier uses "Zaps," automated workflows that link your apps and services. Each Zap has a "Trigger," an event in one app that starts the workflow, and an "Action," a subsequent automated task in another app.

For instance, by setting up a Zapier webhook trigger with Testkube, you can automate an email alert for every failed test case, simplifying how you monitor and respond to testing outcomes. Let's delve into this application next.

Creating a Zap

The first step is to create a Zap. We will create a Zap with a webhook as a trigger and Gmail as an action. This will give us a webhook endpoint that we will use to configure a webhook in Testkube.

Creating a Zap is pretty straightforward, and Zapier gives you multiple options to create a Zap: using AI Zap builder, Pre-built Zap templates, and blank Zaps. In this case, we will create a blank Zap and configure a webhook.

You can follow these steps to create a Zap and configure a webhook to trigger a Zap.

At this point, Zapier will give you a webhook URL, using which you'll create a webhook in Testkube.

Creating a Testkube Webhook

Creating a webhook trigger in Testkube is easy. You can create Webhooks using the dashboard as well as the CLI. We'll be using the dashboard to show you how it works.

Using Dashboard

We'll set up a webhook for Zapier using the dashboard. This webhook will trigger on the test-end-failure event and send the data to the Zapier webhook endpoint that we configured in the previous step.

Navigate to the webhooks section and "Create a new webhook."

Provide a name for your webhook.
Choose the "Resource Identifier" as any test or executor on which you want the webhook to trigger. In this case, we will use a simple curl test. You can find the test in this repo - we've changed the HTTP code in the test so that it fails.
For "Triggered events", choose end-test-failed, which basically means it will trigger when the chosen Pytest fails.

On the next screen, provide the Zapier webhook endpoint URL you generated while setting up Zapier integration. Now, we have a successfully configured Zapier webhook.

Now, when you execute the curl test and when it fails in the Zapier dashboard, you'll see the Testkube records. At this point, you've successfully configured Testkube Webhook to send data whenever the curl test fails.

Click on "Continue with selected record button" to configure the action, in this case, Gmail.

Configuring Gmail

In the action field, choose Gmail and configure your Gmail account. It uses the familiar Google Sign-In, so you simply need to choose your account, provide Zapier the necessary permissions, and your Gmail account will be configured within a few seconds.

Based on the test data that Zapier received, it allows us to insert variables In the various fields. Using those, we'll configure the following:

To: Email address of where the mail will be sent to.
Subject: Provide the subject of the email. We'll provide "Test Name "
Body: Here, we include details like the Test name, Start Time, Duration, and Logs - a link to the test execution.

With this setup, you should have a Zap like the one above. We have configured the email body with details like the test name, start time, duration, and logs, which is the URL for the test execution log.

Testing the Webhook

Now that we have configured the webhook for Zapier, it's time to test it. To do that, simply run the curl test.

When the test fails, you'll see your Zap trigger and the associated action being performed. In this case, it will trigger an email to Gmail.

Based on our configuration, below is the mail we received after the Zap was executed successfully.

That's about it. Creating a webhook in Testkube and integrating it with Zapier to send emails using Gmail is so simple.

In fact, using Zapier you can have endless possibilities as it has integration with virtually every tool and service that you know of. So you can set up a webhook in Testkube and configure Zapier to:

Send test status into an Excel sheet
Create an issue on GitHub
Make an entry about the test failure in Notion
Send a notification to your team on Slack
And much more.

Closing Thoughts

To sum it up, enabling webhooks in Testkube elevates the testing process by offering real-time alerts to various test events. This not only streamlines your testing workflow but also enhances the reliability and efficiency of your testing strategy.

By integrating Testkube with Zapier, you unlock a world of endless possibilities. It allows you to connect Testkube to virtually any service you can think of.

We invite you to explore the power of Testkube and try Webhooks. Join our community on Slack to share insights, seek help, and connect with fellow developers.