Forem: Michal Nowikowski

Kraken CI vs Other CI Systems

Michal Nowikowski — Fri, 23 Jun 2023 05:23:40 +0000

Introduction

In this section, three CI/CD tools will be compared with Kraken CI. These tools are among the most popular systems:

The most significant difference between Kraken CI and these three
systems, as well as other CI/CD systems, is that they lack test result
analytics. All three systems barely touch upon the topic and are only
able to report results acquired during a build.

Having only information about passes and failures, it is difficult to
make a decision. It is not known if this failure has appeared
before. Maybe it has been failing for several months, or perhaps it is
sporadic and only fails from time to time. Alternatively, it could be
a significant regression that requires urgent fixing.

This requires either using other third-party tools that can fill this
gap or developing custom scripts that can analyze the results and
generate comprehensive reports.

In contrast, Kraken CI focuses on this topic and excels in both
execution and results analytics. It can show:

fixes and regressions,
age of test result,
stability of a test case,
importance of negative test result,
history of a test case in a table and in a chart,
total results chart ie. pass ratio over time,
pass ratio chart for selected subset of tests (e.g. tests results obtain on particular operating system).

Furthermore, it allows users to comment on test results, such as
referencing a submitted bug or identifying the root cause, among other
things.

The details are presented in Test Results Basics and Test Results Tracking of Kraken CI docs.

But now let's discuss the differences between specific CI/CD systems.

Jenkins

Jenkins is an open-source on-premises automation server used for building, testing, and deploying software. It offers a broad range of plugins and integrations that support continuous integration and continuous delivery workflows. Jenkins is widely used in software development and is recognized for its flexibility and extensibility.

Bright Side

Jenkins has many powerful features, with one of the most prominent being its flexible and convenient method for defining pipelines using scripting code.

Another very positive aspect of Jenkins is its community. The
community, besides providing lots of blog posts and tutorials about Jenkins, also develops many plugins that greatly improve Jenkins.

Dark Side

1. Plugin Overload and Compatibility Issues

Jenkins relies heavily on plugins for extended functionality. However,
the sheer number of plugins available can make it challenging to
select, manage, and maintain the right set of plugins. Plugin
compatibility and stability issues can also arise, leading often to
conflicts or performance problems.

Kraken CI has a different approach. All of its features are built-in, which ensures compatibility and thorough cross testing. Users are encouraged to contribute directly to the core of Kraken CI. The only extensions available to users are tools, which enable the integration of third-party tools with Kraken CI. This is similar to third-party actions in GitHub Actions.

2. Bad User Experience in UI

Jenkins' user interface (UI) is often criticized for being outdated
and not user-friendly. There is a dichotomy between the classic UI and
the Ocean Blue UI. Both show more or less the same things, but one is
better in some areas, while the other is better in other areas. This
leads to confusion.

Many UI pages in Jenkins lack usability. Some areas are exceptionally
bad, such as presenting logs. When the logs are very large, Jenkins
shows them very slowly, especially if timestamps are
enabled. Sometimes the UI freezes completely and cannot show
anything. There is also a page called Pipeline Steps that shows logs
for each step separately, but there are so many internal steps
presented that it is very hard to navigate.

Presenting logs in Kraken CI is impressive. They are divided by
default into steps that the user has defined in a workflow, or they
are displayed in a flat format. Users can enable internal logs to see
details, and loading logs is paged, so it is not necessary to load the
entire log at once. However, it is still possible to download the
entire log in raw format. Logs can be filtered by multiple fields and
viewed at the individual step level, job level, run, flow, branch, or
project levels. ANSI codes are colored, and there are many more
features. Check out these two blog posts that present these features:

Standing out Logs Handling and Further Logging Improvements.

There are more examples like that. For instance, navigation, browsing
test results, and so on. In Kraken CI, all of these aspects were
thoughtfully reevaluated and engineered with user experience (UX) in
mind.

3. Reliance on Groovy Scripting

Jenkins pipelines, which are used to define CI/CD workflows, are
written in Groovy scripting language. While this provides flexibility
and extensibility, it requires from developers to learn and use Groovy,
which can be an additional learning curve for teams not familiar with
Groovy.

In the case of Kraken CI, Python is used as the language to define workflows. According to the TIOBE Index, Python is ranked number 1, making it the most popular programming language,
while Groovy is ranked 44th.

GitHub Actions

GitHub Actions is a powerful and flexible automation platform that allows you to streamline and optimize your software development workflows. With GitHub Actions, you can automate tasks, build, test, and deploy your applications directly from your GitHub repositories.

Bright Side

The biggest advantage of GH Actions is that it is integrated with
GitHub repositories, and it is free up to a certain usage level. In
addition, it allows for the extension of defined workflows by using
community-developed actions. The number of available third-party
actions is quite substantial.

It is also possible to expand the computing power by connecting your
own "self-hosted" machines to the pool of all machines. Therefore,
workflows can run on both GitHub's machine cluster and on your own
on-premises machines.

Aside from that, GitHub Actions can do more than just CI/CD. Workflows
can be triggered for various project events, such as a new issue being
submitted, a new comment being added to an issue, a repository being
forked, a release being created, and more.

Dark Side

1. Poor, not Scalable UI/UX

The user interface is cluttered and does not scale well for larger
projects. When a workflow reuses other workflows, the job names become
excessively long and are frequently truncated in multiple areas within
the UI. This makes it difficult to identify which jobs we are
currently viewing. The problem is further compounded when nested
workflows are utilized.

The UI also does not scale well for a larger number of jobs. If there
are more than 30 jobs in a workflow, it becomes impossible to filter,
sort, or list them in pages. Instead, all jobs are just presented to
you in a flat list.

One more weakness is the logs panel. Initially, it appears to be
sleek. Nonetheless, if there are numerous steps and the logs are quite
extensive, containing thousands of lines, then navigation becomes
difficult. While jobs are in progress, the UI attempts to predict what
to display, causing it to jump from one location to another, making it
difficult to choose the correct step. Additionally, paging through the
logs is not feasible. They are all displayed at once and are
progressively loaded while scrolling down. This is a pretty
ineffective method of finding the appropriate location in the log if
the log contains hundreds of pages.

Kraken's logging facilities in the UI were already described in the
Jenkins section. In terms of UI scalability, this is one of Kraken's
goals. It is designed from the ground up to handle multiple jobs, up
to thousands or even tens of thousands. It provides means for
filtering and sorting jobs, and presenting workflows, even those that
are more complicated and involve multiple forks, in a more scalable
way than GH Actions.

2. Reusability Is Quite Hard

Initially, it seems that reusability is well supported in GH Actions
workflow using third-party actions and reusable workflows. However,
upon closer inspection, it is not as seamless as it
appears. Externalizing some parts of the workflow into reusable
actions or workflows can be quite tedious. If there is a need to
locally reuse a fragment of code, one would like to use the YAML
feature called anchors and aliases. Unfortunately, GH Actions does not
support this feature.

The situation becomes complicated if a user needs to maintain multiple
repositories. Certain parts of workflows can be reused, but the
framework of the workflow, including trigger definitions, global
environment variables, and main jobs (which may reference reusable
workflows), must be present in every repository. Ideally, one would
like to fully externalize the entire workflow to a single repository
and simply reference it in every other repository.

Kraken allows for defining a workflow for particular repository in
several way. The workflow definition can be stored

in Kraken server,
in the same repository,
in another repository.

In the first case, the entire workflow definition resides in the
Kraken server's database. It is then possible to link the workflow
definition to any Git repository - whether it is the repository that
needs to be built and tested or any other repository. This makes it
easy to experiment with a workflow for any repository without the need
to make changes. Later on, it is easy to transfer that workflow to the
relevant repository. Furthermore, the entire workflow definition can
be extracted to a separate repository and reused in multiple other
repositories.

3. Cloud and Security

GitHub Actions is a cloud-based CI/CD tool, which means that its
reliability and availability are out of your control. If an outage
occurs, the only thing left for you to do is wait until GitHub's team
fixes their services. This may last from a few minutes to a few hours.

Although GitHub Actions provides security features like access
controls, secrets management, and vulnerability scanning, it may not
fulfill the specific security or compliance requirements of all
organizations or industries. Organizations with stringent security or
compliance needs must evaluate GitHub Actions' security features
carefully and ensure that they align with their requirements.

GitLab CI

GitLab CI is a comprehensive DevOps platform that combines source code management, CI/CD pipelines, issue tracking, and more in a single
integrated solution. It enables organizations to streamline their
software development processes, promote collaboration, and achieve
faster and more efficient delivery of high-quality software.

Bright Side

GitLab CI enables the definition of pipelines as code by using a
YAML configuration file. This approach makes the CI/CD process
version-controlled, easily replicable, and auditable. Infrastructure
modifications, test scripts, and deployment steps can be defined
alongside the code, promoting transparency and collaboration.

Furthermore, GitLab is offered as software that can be installed
on-premises, as well as a service that can be used in the cloud. This
way, it combines the software aspect of Jenkins with the cloud service
aspect of GitHub Actions in one solution. The choice belongs to the user.

GitLab offers a wide range of features and integrations, with most of
them only available in the Enterprise Edition. These features include
code quality checks. Another strong aspect of GitLab is its
reusability, as it is easy to reuse fragments of YAML definitions
thanks to anchors in the language.

Dark Side

1. Too complex in advanced cases

The learning curve for GitLab CI/CD is rather steep, and some poor
design decisions have made configuring CI/CD pipelines in the
.gitlab-ci.yml file quite complex. The documentation is full of
notes that certain aspects are not obvious, discouraged, or may
behave unexpectedly. Furthermore, the issue tracker is filled with
problems related to excessively complicated features.

One of the problems is that a CI pipeline is triggered by all possible
events. Then it is user role to explicitly exclude which jobs should
be run or not for particular event. Sometimes it leads to situations
that a pipeline is triggered multiple times. There is a dedicated
section in the docs how to avoid such cases:
https://docs.gitlab.com/ee/ci/jobs/job_control.html#avoid-duplicate-pipelines

All of these factors make developing more advanced workflows quite
challenging.

In contrast, Kraken is more deterministic. Workflows are triggered by
user-indicated events, and it is clear why a given workflow was
started - this information is presented in the UI.

Hello Worlds in 22 Langs on Kraken CI

Michal Nowikowski — Fri, 13 May 2022 04:45:04 +0000

There is a new, quite funny workflow example for Kraken CI. It shows how to compile and run Hello World programs in various programming languages, 22 languages.

Hello World Programs

Every programmer knows what does Hello World program looks like.

Here is an example in C language:

#include<stdio.h>

int main(){
  printf("Hello World\n");
  return 0;
}

But 22? These 22 can be seen in Kraken's repo on GitHub:
https://github.com/kraken-CI/hello-worlds

Still, there are more languages with Hello Worlds.
The biggest collection can be found here:
http://helloworldcollection.de/.

These are just examples. In our case, they can be compiled and run. After downloading the repo you may run ./run.sh script providing the language folder, e.g.:

$ ./run.sh C

This will display, compile and run a C example.

In Kraken's case, there is defined a workflow that does it for us for each language. But first, we need to have systems with a compiler or a runtime for these languages. Docker to the rescue.

Kraken Base Language Images

For each language, there is prepared a Docker image. They are based on official images for these languages. For Kraken, they got some extras like Kraken Agent that is running using kraken user (not root).

The list of all lang images is available in the docs.

The Workflow

The mentioned workflow can be found in kraken-workflow.py file.

In short, it looks as follows:

def stage(ctx):
    langs = [
        ('C', 'gcc:11.3'),
        ('C++', 'gcc:11.3'),
        ('C#', 'mono:6.12'),
        # and more...
    ]

    jobs = []

    for lang_name, lang_image in langs:
        jobs.append({
            "name": "hello world %s" % lang_name,
            "steps": [{
            "tool": "git",
                "checkout": "https://github.com/Kraken-CI/hello-worlds.git",
                "branch": "main"
            }, {
                "tool": "shell",
                "cmd": "./run.sh %s" % lang_name,
                "cwd": "hello-worlds"
            }],
            "environments": [{
                "executor": "docker",
                "system": "krakenci/%s" % lang_image,
                "agents_group": "all",
                "config": "default"
            }]
        })

    return {
        "parent": "root",
        "triggers": {
            "parent": True,
        },
        "parameters": [],
        "configs": [],
        "jobs": jobs
    }

As you may notice, the list of jobs is built dynamically. For each entry of the list of tuples (lang-name, docker-image), a job is defined with the same steps but various values for language and Docker image name. The lang name is passed to ./run.sh script in shell step. And Docker image name is used in the environments section, in the system field. Such a list of jobs is then returned in the whole workflow definition. Each job will be run in parallel in separate Docker containers.

The example of workflow execution is present in Kraken lab:
https://lab.kraken.ci/runs/5009/jobs. Check the workflow definitions in branch management page.

This screenshot shows the execution of this workflow:

Summary

This article shows how Kraken CI can build programs written in various languages. Kraken provides prepared Docker images for many languages. It is easy to define sophisticated workflows that dynamically define jobs that can be run in parallel. In the end, it is nice to observe the results of workflow execution in Kraken UI.

MLOps with MLflow on Kraken CI

Michal Nowikowski — Fri, 29 Apr 2022 11:55:24 +0000

Besides building, testing and deploying, Kraken CI is also a pretty nice tool to build an MLOps pipeline. In this article, it will be shown how to leverage Kraken CI to build a CI workflow for machine learning using MLflow.

Kraken CI is a new Continuous Integration tool. It is a modern, open-source, on-premise CI/CD system that is highly scalable and focused on testing. It is licensed under Apache 2.0 license. Its source code is available on Kraken CI GitHub page.

MLOps and MLflow

MLOps is a set of practices that aims to build and maintain machine learning models in production reliably and efficiently. One of prominent tools in this area is MLflow.

MLflow is an open-source platform for managing the end-to-end machine learning lifecycle. It tackles four primary functions:

Tracking experiments to record and compare parameters and results (MLflow Tracking).
Packaging ML code in a reusable, reproducible form to share with other data scientists or transfer to production (MLflow Projects).
Managing and deploying models from various ML libraries to a variety of model serving and inference platforms (MLflow Models).
Providing a central model store to collaboratively manage the entire lifecycle of an MLflow Model, including model versioning, stage transitions, and annotations (MLflow Model Registry).

MLflow in Kraken CI

In the following sections, I will describe how to prepare a workflow in Kraken CI to train an ML model. This is an LSTM model that will predict stock prices based on historical data.

The workflow will be:

pulling live stock data and preparing it for training (source 1,
source 2)
performing the training (source 3)
storing model metrics in Kraken CI for charting

The MLflow project is described in MLproject.

Workflow Definition

The whole Kraken CI workload is defined here.

There are 3 steps:

   "steps": [{
       "tool": "git",
       "checkout": "https://github.com/Kraken-CI/mlflow-example.git"
   }, {
       "tool": "shell",
       "cmd": "/opt/conda/bin/mlflow run .",
       "cwd": "mlflow-example",
       "timeout": 1200
   }, {
       "tool": "values_collect",
       "files": [{
           "name": "metrics.json",
           "namespace": "metrics"
       }, {
           "name": "params.json",
           "namespace": "params"
       }],
       "cwd": "mlflow-example"
   }],

Checkout mflow example project sources
Run the mlflow project ie. download data, prepare it, run a training and at the end store metrics about the trained model to metrics.json
Upload collected metrics together with hyperparameters from params.json to Kraken server

The last step allows for charting accuracy and RMS of the model over builds.

There is one more element defined in the workflow: the definition of execution environment:

   "environments": [{
       "system": "krakenci/mlflow",
       "executor": "docker",
       "agents_group": "all",
       "config": "default"
   }]

Here we can notice the use of a pre-prepared image with mlflow. It is available in Docker hub: krakenci/mlflow.

The whole example of workflow is present in Kraken lab:
https://lab.kraken.ci/branches/32/ci. Check the steps definitions in
branch management page.

Execution and Monitoring

Besides the workflow definition, Kraken UI also shows collected data
and the charts drawn from this data:
https://lab.kraken.ci/test_case_results/595950, the charts tab.

The right chart shows value of loss collected over time:

Summary

This article shows how Kraken CI can be used to build an MLOps pipeline. The pipeline downloads raw data, prepares the data for training and then executes the training. The trained model metrics are collected and charted in Kraken UI at the end.

What is Wrong with Your Testing?

Michal Nowikowski — Mon, 21 Feb 2022 06:33:22 +0000

As I observe, software projects mostly have nearly all the time 100% pass ratio. But I think this is not possible in reality, and we cheat ourselves. This may be due to a few reasons:

a project was not that big, so it was not that hard to achieve 100%
lots of failing or problematic tests were marked as skip, so they are not executed
there was not many tests or no tests at all, i.e. tests coverage was non-existing

Generally, it is impossible to have a 100% pass ratio in more significant projects. If we look at a bug database, we find plenty of various issues. If there were tests that reveal them, then they would be failing.
Maybe this is due to poor tools for tracking many tests and their results. This is an area where Kraken CI can help.

Introduction

ISTQB defines several goals for testing. Two
of them that are pretty important are:

To find defects and failures thus reduce the level of risk of inadequate software quality.
To provide sufficient information to stakeholders to allow them to make informed decisions, especially regarding the level of quality of the test object.

At first glance, these goals do not seem that hard. In small projects,
it is pretty easy to achieve. Not many tests and not many code changes
do not generate much work to analyze and reason.

In the case of bigger projects, where testing is performed even several
times a day, and where test content contains thousands or tens of
thousands of test cases and where the number of combinations of
environments (like operating systems and various types of hardware)
and configurations is high, tracking and monitoring tests results
gets really difficult.

Proper tools must be employed to match these expectations that will
automatically do much of this analytical work. Basic tests results
reporting is not enough. More sophisticated analysis is required. The
following sections present tools and features in Kraken CI that help in
this area.

Result Changes in the Last Flow

The basic approach to testing is observing results. The results
include passes, failures, errors and sometimes some other types of
status. Looking at failures and errors allows for finding some
problems, these lead to some debugging, root-casing, defect
submission and fixing eventually.

Observing a list of results from subsequent test sessions for
subsequent product builds is not easy anymore. Again, there are some
failures, but which were also present in the previous build and which
are new regressions? In the case of small projects with a handful of
tests, it is easy to go back to previous results and compare and
manually find out what has changed. If there are tens or hundreds of
failures, then a feature is needed to spot regression or fixes
compared to the previous build.

New button

In the case of Kraken CI, the table with test case results allows
immediately for filtering and showing only these results that are
regressions or fixes. For that purpose, a New button shows only the
test cases that changed the status in the current build compared to
the previous one, i.e. shows "new", interesting results that require
some attention.

The screenshot above shows the New button and filtered results list.
The two first test case results indicate a regression and a fix.
This way, we can focus only on results that indicate some changes and
ignore the failures that are there for weeks or months because they
were not fixed yet.

Filter by Result Change

It is possible to do more fine-grain filtering and select a particular
change, e.g., regression or fix. For this purpose, there is a
dropdown Result Change.

The screenshot shows selecting regressions only. So now we can focus
only on a negative aspect of result changes.

Instability

Even if only the regressions are presented, some tests are still not
interesting or maybe interesting but for different reasons. These not
interesting results are sporadic or unstable tests. Sometimes a test
case passes, sometimes it fails. It changes its results status quite
often. It can be concluded that this is all the time the same problem:
either there is a bug in the product or in the test. If this is a
known problem, it is better to filter out such results. This is
possible by filtering results by their instability. There is a column
called Ins. (for instability) that shows a count of status changes
in the last 10 builds. If it is equal to 9 it means that it was
flipping from passed to failed, back and forth. So using a slider in
the filtering pane, the instability can be narrowed to, e.g. 0-3
range.

This screenshot shows results filtered by instability that is set to
range 0-3. Now it is visible in the Instability column that it is 1
i.e. the regression is new, and in the past 10 builds, the test cases
were stable and passed.

In the previous case and previous screenshot, this column was showing 9. Let's get back to this situation and click such a test case to check its details.

This screenshot presents a table with the historical results of the
selected test case. At the top, there is the latest result in the
first row. The Result column shows result statuses. It is visible
now that the status was constantly changing in the last results. The
Age column shows 0, i.e. every time the result was new. The
Instability column is all the time high. Such test case require some
investigation and either fixing or removing from the test scope.

It is possible to switch the view to a chart by selecting the second
tab at the top of the page: Charts. Here it is also visible that the
results are pretty unstable.

Relevancy

Another way for getting important results is sorting them by
relevancy. This is a metric built on properties of particular
results. The higher the relevancy is, the more important the result
is.

This relevancy score is calculated as follows:

+1 if the result is not passed
+1 if there is no comment attached to the result ie. it is not root-caused yet
+1 if the result is failure
+1 if the result was pretty stable (so it is the same over several test runs), ie. instability is <= 3
+1 if the result is rather recent (the age is < 5) ie. ignore old failures
+1 if this is a regression

Max relevancy is 6. So more interesting results are recent
regressions that till now were pretty stable and are not root-caused
yet.

Comments

To remember the conclusions of investigation of given failure it is
good to store them. For that purpose, there are test case comments.

There are two columns on the right side of the results table related
to comments. Clicking a given cell allows for entering a comment a
state. The state may indicate if the problem has been root-cased or
not. This way, the next we get the results for this same test case, we
will already see the comment and conclusions we made previously.

Finding Changes in the Past

It is hard to spot regressions if the results are not analyzed
immediately after tests execution. The failures are getting stable, so
it is not visible as regressions anymore. To find such results,
results that were regressions during, e.g. last week, we need to
filter by age. Age allows filtering results that are not recent but
happened in the near past. If, e.g. we set age to 1-5 range, then the
test case results table will show results that persist (are stable in
the last 1-5 runs) the change happened in the previous 1-5
executions. This allows for filtering out recent failures (because
e.g. we already have analyzed them) and very old, stable results
(because e.g. no one cares about them anymore).

This screen shows results filtered by age from 2 (since 2 builds ago)
to let's say infinity. The table at the bottom contains now only a
handful of results with age 2 or 3. This way, we can find not current
regressions but still quite recent.

Tests Basics in Kraken CI

Michal Nowikowski — Mon, 24 Jan 2022 16:14:31 +0000

In the last few articles there was a lot about autoscaling. Now let's get back to the core of Kraken CI: testing.

Introduction

Testing in Kraken CI is much more organized that it is in e.g. Jenkins. Testing is planned in branches and high level view on results is also presented for branches. Then, it is possible to dig deeper into runs, jobs and particular test case results. More details about all this can be found in the following sections.

Branch Results

The Branch Results page presents high-level information about the flows and runs of a given branch. It organizes flows in rows. The top row contains the latest flow. Each flow row contains a list of boxes. Each box contains information about a run of a given stage. The order of boxes with runs is chronological according to their start time.

Run Box

The color of the run box indicates the final status of the run:

red: one or more jobs finished with an error
yellow: no errors in jobs but some test results failed or there are issues discovered in the code
green: all jobs finished without error and there are no failures in tests and no issues in the code

A run box contains:

data about execution: total number of scheduled jobs, number of erred jobs, etc.
summary of testing: pass ratio, total, regressions and fixes counts, etc
issues summary: total number of discovered issues, number of new issues

Example 1:

This run has 1 successful job that reported tests results. There were 100 test cases run, where 16 did not pass, so the pass ratio is 16%. There were detected 15 regressions and 13 fixes. There are failures, so the box is yellow.

Example 2:

This run has 4 jobs with no errors. Some of them reported tests results, the other ones reported issues. There were 17 test cases run, all passed, so the pass ratio is 100%. There were detected 15 issues in the code. There are issues, so the box is yellow.

Example 3:

This run has 2 jobs with no errors. In the reported tests results, there are 100% passed test cases. No failures and no issues, so the box is green.

Example 4:

This run has 4 jobs with 1 error. All tests reported are passed but there could be not reported tests due to errors in job execution. There were detected 6 issues in the code. There is one erred job so the box is red.

Test Case Results Table

A job that performs testing reports test results. The test results are visible on a Run Results page, in Test Results tab. They are presented in a table.

The table presents test case results in each row. They contain details of the test case result:

the result status
a name of the test case
job name and ID
age, instability and relevancy
values
execution environment: system, agent group and actual agent and config

In Kraken, a test case result besides status can contain values. Values are pairs: attribute and its value. They are also a result of a test case but hold numbers. They can hold e.g. performance data or anything similar to performance. These values are tracked over time. Kraken can draw charts of values and do more things with them.

Filtering Test Case Results

It is possible to filter test results by various attributes:

by Results Status ie. Passed, Failed, etc.
by Result Change ie. Fixes, Regressions, New and No Changes; the change is determined between the current result and the previous result from previous run
by Age where age is a number of previous runs with the same result as in the current run; so e.g. age 0 means that in the previous run test case had a different result than it has now (so these are regressions and fixes); age 1 means that the current run and the previous run has the same result but the result before the previous one is different
New is a shortcut to set Age between 0 and 0, ie. show changes that happened in the last run (fixes + regressions)
by Instability where instability is a count of result status changes in the last 10 runs; if this is 0 then in the last 10 runs test case has the same result (it was pretty stable); if this is 10 then it means that in every run of the last 10 runs the status was changing compared to the previous one (it was very unstable)
by Test Case name
by Job name of ID

Example 1:

Filtering by Result Changes: show only regressions. Regression means a change from non-passed status to passed status.

Example 2:

Filtering by Result Status and Result Changes: show only failures that are regressions.

Example 3:

Filtering by Age: show only results in age between 2 and 3.

Example 4:

Filtering by Instability: show only results that have instability greater than 7 i.e. very unstable test cases.

Sorting Test Case Results

Test case results can be sorted. This can be done using several columns:

name
result
change
age
instability
relevancy

By default results are sorted by test case name.

One of the most important sorting order is by Relevancy.
Relevancy is estimated according to this rules:

0 is initial
+1 for not passed
+1 for not root caused problem
+1 for failure
+1 for instability <= 3
+1 for age < 5
+1 for regression

This way the most relevant results get the highest score. They need an
attention.

Comments

In the test case results table, it is possible to add sticky comments to test case results. When a new failure is spotted, a comment stating, e.g. the reason for the failure or link to a bug in a bug tracker may be added.

To add a comment, click on the cell of a particular test case result in the Comment or Comment State columns.

This will reveal a dialog box that shows a form for introducing new comment and a history of previous comments related to this test case.

Each comment has a state. It can be one of:

new
investigating
bug in product
bug in test

The state allows concluding quickly if given failure has been analyzed and it is known where is the root cause and what needs to be fixed or if the failure still requires some investigation.

If a failure is root-caused, then its relevancy is decreased by 1.

If the failure persists in the subsequent test runs, the added comment is still presented next to the new results. If new facts about the failure are revealed, you may extend the comment.

Test Case History

Each test case has its history of results. In the table with test case results a link under the test case name leads to a page dedicated to the given test case. This page presents a table with the historical results of the test case.

There is another tab called Charts that shows these results on charts.

The left chart shows the history of result status. The right one presents the history of values in time. Error bars are shown if there were run multiple iterations and there were reported multiple values for a given attribute.

Autoscaling CI on Kubernetes in Kraken CI

Michal Nowikowski — Thu, 30 Dec 2021 14:04:44 +0000

This tutorial is the fifth installment of the series of articles about Kraken CI. Part 1, Kraken CI, New Kid on the CI block, presented the installation of Kraken. The second part covered how to prepare a workflow for a simple Python project. The 3rd one was about autoscaling on AWS and Azure. The last part was introducing to Webhooks in Kraken CI.

This time we would like to show the latest feature that was developed in Kraken CI: deploying Kraken CI and autoscaling in Kubernetes.

Intro

One of the methods of deploying Kraken CI is installing it into Kubernetes cluster. Kraken CI is natively divided into several services packed in Docker images so they can be nicely laid out in the cluster. The other aspect is running Kraken jobs. In this case, they are run in containers natively scheduled onto Kubernetes nodes.

This guide shows how to install and configure Kraken CI in Kubernetes to leverage its potential.

Pre-requisites

Several things are required to install Kraken CI in Kubernetes using Helm. In short:

Kubernetes cluster and kubectl tool
Helm tool

Helm is used to deploy several Kraken services and expose them to an external network. These Kraken services are described in Architecture chapter.

Kubernetes Clusters

There are multiple ways for setting up a Kubernetes cluster. One of the easiest ones that is most often used for experimenting is Minikube. There are also managed clusters like EKS (Elastic Kubernetes Service) in AWS.

This manual will show how to install Kraken CI in Minikube but the steps are similar for other Kubernetes environments as well.

Install in Minikube

First, download minikube from https://minikube.sigs.k8s.io/docs/.

And then create a cluster:

$ minikube start

Now you may install Kraken CI, but first, let's add a repo with Kraken Helm charts:

$ helm repo add kraken-ci https://kraken.ci/helm-repo/charts
$ helm repo update

and now install Kraken CI:

$ helm upgrade --install --create-namespace --namespace kraken \
  --debug --wait \
  --set access.method='external-ips' \
  --set access.external_ips={`minikube ip`} \
  kraken-ci kraken-ci/kraken-ci

This command actually upgrades Kraken CI if it is already installed but if it was not yet installed, then it installs it.

More details about using Helm to install Kraken CI can be found in installation docs.

When everything completes successfully, then at the end of the whole output there should be presented short instruction about getting the URL of Kraken service like that:

NOTES:
Get the application URL by running these commands:
  export NODE_PORT=$(kubectl get --namespace kk-1 -o jsonpath="{.spec.ports[0].port}" services ui)
  export NODE_IP=$(kubectl get nodes --namespace kk-1 -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT

Now you may check if Kraken is working by visiting the URL
given by this code and by checking if Kubernetes is running Kraken's services with this command:

kubectl get all -n kraken

This will show Kraken's pods, services, deployments and replica sets.

At this moment, Kraken CI should be operating. Please visit the website on the URL presented above. You should see a login page:

Enter admin for Username and admin for Password.

Global Settings

First, let's set some global settings. In Web UI, click Settings in the top menu bar.

In the General tab, fill the following fields:

Kraken Server URL - an URL with a port of the server that is visible inside the Kubernetes cluster
MinIO/S3 Address - the IP address and port of Kraken's internal artifacts storage address
Clickhouse Proxy Address - the IP address and port of Kraken's internal logs storage address

In all these cases minikube IP address can be used. It can be obtained using minikube ip command.

Having set the General settings, let's move to the Cloud tab.

Here, please move to Kubernetes section and set only the Namespace field to the value that was provided in helm command above (if it was not changed, then it should be kraken).

API Server URL field must be empty as Kraken CI is installed inside Kubernetes and it knows where the API server is.

After saving the settings, check if connectivity is ok using Test Kubernetes Access button.

Configuration in Agents Groups

After setting global settings, it is possible now to configure aspects of spawning Kubernetes pods for Kraken jobs. This can be done on Kraken -> Agents -> Groups page. Let's create a new Agents Group by clicking Add New Group button and naming it k8s. The newly created group's details will be presented on a separate tab. On this tab, there is a section Agents Deployment - select Kubernetes. In this
case, there is only one field to set: Instances Limit.

This limit value will not allow having more running pods than it - here 5.

Job Definition

Now, to use the defined k8s Agents Group, we need to prepare a project with a branch and a stage. More details about that can be found in Introductory Guide. So let's concentrate now on defining a job.

{
    "parent": "root",
    "triggers": {
        "parent": True
    },
    "configs": [],
    "jobs": [{
        "name": "hello",
        "timeout": 500,
        "steps": [{
            "tool": "shell",
            "cmd": "echo 'hello world'"
        }],
        "environments": [{
            "system": "ubuntu:20.04",
            "agents_group": "k8s",
            "config": "default"
        }]
    }]
}

There is not much difference compared to regular Kraken jobs. The job has a defined environments section where we are pointing to our k8s Agents Group. In the case of system field here, we have a Ubuntu:20.04 Docker image.

Run

Now when a job is assigned to an agents group with configured Agents Deployment then a new pod will be spawned for that job if agents are not available in the Kraken.

Let's change the view to Branch Results view and trigger a new flow by clicking Run Flow button. On the run page, the list of jobs shows our AWS job:

That's it!

Webhooks in Kraken CI for GitHub, GitLab and Gitea

Michal Nowikowski — Mon, 15 Nov 2021 11:49:39 +0000

This tutorial is the third installment of the series of articles about Kraken CI. Part 1, Kraken CI, New Kid on the CI block, presented the installation of Kraken. The second part covered how to prepare a workflow for a simple Python project. The last one was about autoscaling on AWS and Azure.

This time we would like to show the latest feature that was developed in Kraken CI: webhooks for GitHub, GitLab and Gitea.

Intro to Webhooks

Kraken CI allows triggering a flow of a branch in a project using a
webhook in a Git hosting service. A push to a regular branch e.g. main
branch or to a branch associated with a pull or a merge request will cause
a Git hosting service to call a webhook exposed by Kraken CI. To make
it happen a webhook URL and a secret have to be stored in a project
settings in Git hosting service.

Currenty there are support the following Git hosting services:

GitHub
GitLab
Gitea

The following guide shows how to configure webhooks: 1) how to enable
them in a project in Kraken CI and then 2) how to set a webhook URL and
a secret in Git hosting service. In the end, the usage of webhooks
will be presented by doing a push to Git repository.

Enable Webhooks in a Project

In Kraken CI go to your project page and switch to WebHooks
tab. There are available webhooks for several Git hosting services.
Enable the one that you are using for hosting your Git repository.
Enabled webhooks show an actuall webhook URL and a secret. These
information should be copied and set in you webhooks setting page in
your Git hosting service.

Set Webhook URL in Git Hosting Service

The following sections show how to set webhook URL to Kraken CI, a
secret that is used to authenticate and they show which event types
should be selected that will trigger a new flow in you Kraken CI
project.

GitHub

The arrows on the picture above indicate where the webhook URL and the
secret from your Kraken CI project should be pasted on your GitHub
repository webhooks settings page. In case of GitHub two event types
should be selected:

Pull Requests - they will trigger DEV flows in you Kraken CI project, in a branch indicated in the event
Pushes - they will trigger CI flows in you Kraken CI project, in a branch indicated in the event

GitLab

The arrows on the picture above indicate where the webhook URL and the
secret from your Kraken CI project should be pasted on your GitLab
repository webhooks settings page. In case of GitLab two event
types should be selected:

Push events - they will trigger CI flows in you Kraken CI project, in a branch indicated in the event
Merge Request events - they will trigger DEV flows in you Kraken CI project, in a branch indicated in the event

Gitea

The arrows on the picture above indicate where the webhook URL and the
secret from your Kraken CI project should be pasted on your Gitea
repository webhooks settings page. In case of GitLab two event
types should be selected:

Push - they will trigger CI flows in you Kraken CI project, in a branch indicated in the event
Pull Request and Pull Request Synchronized - they will trigger DEV flows in you Kraken CI project, in a branch indicated in the event

Trigger a Flow

Now, in your a folder with source code repository you may invoke git push command:

$ git push

This should trigger a new flow in CI branch in you Kraken CI project.

If you create a new branch, do some commits to it, then push this new
branch, and then create a pull request (PR) or merge request (MR) in
your Git hosting service UI, then all this will trigger a new DEV flow
that is a base branch for the PR or MR:

$ git checkout -b my-branch
$ git push --set-upstream origin my-branch

That's it!

Autoscaling CI with Kraken CI

Michal Nowikowski — Mon, 19 Jul 2021 05:41:40 +0000

This time we would like to show the latest feature that was developed in Kraken CI: autoscaling.

Intro

Maintaining a fixed number of machines for Kraken CI can be costly. If there is no high traffic of builds, then some machines sit idle and waste power. The solution to this problem is autoscaling ie. spawning machines with Kraken Agents dynamically depending on the current needs in the Kraken CI. If there is submitted a new flow with many building or testing jobs, then new machines are spawned. When the flow completes, machines are terminated and no costs are incurred.

Currently, Kraken CI autoscaling is supporting AWS EC2 virtual machines.

The following chapters show how to configure access to AWS, then how to configure parameters and behaviour of spawned machines. In the end, the usage in job definitions is presented.

Details about configuration can be found in the Kraken docs.

Global Cloud Settings

First, global settings have to be set to allow access to a given cloud provider. In Web UI, on Kraken -> Settings page, in Cloud tab, there is a form for collecting credentials to cloud providers:

In the case of AWS, there are required Access Key and Secret Access Key. After filling in the values and saving settings, it is possible to check if it works by clicking Test Access button.

Preparing Cloud Environment

In the case of AWS it is also required to assign proper permissions so that Kraken can create or destroy EC2 instances. The list of all requires permissions looks as follows:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:CreateKeyPair",
                "ec2:CreateSecurityGroup",
                "ec2:CreateTags",
                "ec2:DeleteKeyPair",
                "ec2:DescribeInstances",
                "ec2:DescribeInstanceStatus"
                "ec2:DescribeInstanceTypeOfferings",
                "ec2:DescribeRegions",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeVpcs",
                "ec2:RunInstances",
                "ec2:TerminateInstances",
            ],
            "Resource": "*"
        }
    ]
}

Configuration in Agents Groups

Having set credentials to cloud providers, it is possible now to configure aspects of spawning new machines. This can be done on Kraken -> Agents -> Groups page. Let's create a new Agents Group by clicking Add New Group button and naming it aws-t2-micro. The newly created details will be presented on a separate tab. On this tab, there is a section Agents Deployment. So the deployment can be manual (default)
or automated to a particular cloud provider. In the case of Amazon Web Services, there are the following options that can be set:

Select any region that fits you. Limit of instances can be set to 2. This will not allow having more running machines than 2. For now, the default AMI can be empty.

Then in VM options, let's stick to AWS free tier and select t2.micro instance type. The rest of the checkboxes leave empty. Init script also can be blank.

To ensure that dangling instances do not eat up our budget, let's set the options in Destruction section to 1 job and 10 minutes. So a spawned machine will be terminated after executing 1 job. And even then, if it sits idle for more than 10 minutes, it will be destroyed as well.

Job Definition

Now, to use the defined aws-t2-micro Agents Group, we need to prepare a project with a branch and a stage. More details about that can be found in Introductory Guide. So let's concentrate now on defining a job.

{
    "parent": "root",
    "triggers": {
        "parent": True
    },
    "configs": [],
    "jobs": [{
        "name": "hello",
        "timeout": 500,
        "steps": [{
            "tool": "shell",
            "cmd": "echo 'hello world'"
        }],
        "environments": [{
            "system": "ami-0967f290f3533e5a8",
            "agents_group": "aws-t2-micro",
            "config": "default"
        }]
    }]
}

There is no much difference comparing to regular Kraken jobs. The job has a defined environments section where we are pointing to out aws-t2-micro Agents Group. What's is new is system field which points to AMI identifier - this indicates AWS EC2 image.

Run

Now when a job is assigned to an agents group with configured Agents Deployment then a new machine will be spawned for that job if agents are no available in the Kraken.

Let's change the view to Branch Results view and trigger a new flow by clicking Run Flow button. On the run page, the list of jobs shows our AWS job:

That's it!

Your First Workflow in Kraken CI

Michal Nowikowski — Thu, 17 Jun 2021 04:12:32 +0000

This tutorial is the second installment of the series of articles about Kraken CI. Part 1, Kraken CI, New Kid on the CI block, presented the installation of Kraken. This time we will cover how to prepare a workflow for a simple Python project. The workflow will:

fetch sources from GitHub,
run tests using Pytest,
build a wheel package,
upload built package to Kraken's artifacts storage.

Here we are using fresh Kraken installation using Docker Compose with built-in Kraken Agent running in Docker container. Installation procedure details can be found in part 1 or in the installation manual from https://kraken.ci project web page.

Project Preparation

Details about a new project setup in Kraken CI can be found in the previous article, part 1 as well.

Here, in short, let's do:

Create a project with a name: Demo
Create a branch with a name: main
Use a pre-created stage and rename it to: Hello World

Stage Hello World

The stage should look as follows:

And the original workflow schema code looks as follows:

def stage(ctx):
    return {
        "parent": "root",
        "triggers": {
            "parent": True,
        },
        "parameters": [],
        "configs": [],
        "jobs": [{
            "name": "hello",
            "steps": [{
                "tool": "shell",
                "cmd": "echo 'hello world'"
            }],
            "environments": [{
                "system": "any",
                "agents_group": "all",
                "config": "default"
            }]
        }]
    }

Now let's try if it works. Click Run CI Flow button.

Git Repo Checkout Step

Ok, let's start extending our schema to make it more real life. First, let's add a step for checking out a Git repository:

"steps": [{
    "tool": "git",
    "checkout": "https://github.com/Kraken-CI/sample-project-python.git"
}, {
    "tool": "shell",
    "cmd": "ls -al"
}],

Now run new CI flow. The output should look like this:

The logs show that the repository has been cloned and ls -al shows sample-project-python folder.

Run Tests Step

Now let's replace the step with listing the current directory with something more interesting, like running tests. Kraken provides several built-in tools for running tests. In the case of our Python sample project, we will use pytest built-in tool.

"steps": [{
    "tool": "git",
    "checkout": "https://github.com/Kraken-CI/sample-project-python.git"
 }, {
    "tool": "pytest",
    "cwd": "sample-project-python",
    "params": "-vv",
    "pythonpath": "src"
 }],

Change the steps in schema editor, save it and run new CI flow. And the job output should be:

Building Step

So if tests succeeded we could build a package of our Python project. Add step for building a wheel package using project's setup.py script:

"steps": [{
    "tool": "git",
    "checkout": "https://github.com/Kraken-CI/sample-project-python.git"
 }, {
    "tool": "pytest",
    "cwd": "sample-project-python",
    "params": "-vv",
    "pythonpath": "src"
 }, {
    "tool": "shell",
    "cwd": "sample-project-python",
    "cmd": "python3 setup.py sdist bdist_wheel"
 }],

Now run CI flow again. The output should look as follows:

It seems that there is missing a package that provides wheel support for Python. Let's install it.

"steps": [{
    "tool": "git",
    "checkout": "https://github.com/Kraken-CI/sample-project-python.git"
 }, {
    "tool": "pytest",
    "cwd": "sample-project-python",
    "params": "-vv",
    "pythonpath": "src"
 }, {
    "tool": "shell",
    "cmd": "sudo apt update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends python3-wheel python3-setuptools"
 }, {
    "tool": "shell",
    "cwd": "sample-project-python",
    "cmd": "python3 setup.py sdist bdist_wheel"
 }],

New output with building:

Storing Artifacts Step

The results of building the sample project are two packages:

dist/sampleproject-2.0.0-py3-none-any.whl
dist/sampleproject-2.0.0.tar.gz

We could store it Kraken storage and make it available for user for downloading. For that purpose there is another built-in tool: artifacts.

"steps": [{
    "tool": "git",
    "checkout": "https://github.com/Kraken-CI/sample-project-python.git"
}, {
    "tool": "pytest",
    "cwd": "sample-project-python",
    "params": "-vv",
    "pythonpath": "src"
 }, {
    "tool": "shell",
    "cmd": "sudo apt update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y - no-install-recommends python3-wheel python3-setuptools"
 }, {
    "tool": "shell",
    "cwd": "sample-project-python",
    "cmd": "python3 setup.py sdist bdist_wheel"
 }, {
    "tool": "artifacts",
    "action": "upload",
    "cwd": "sample-project-python/dist",
    "source": [
        "sampleproject-2.0.0-py3-none-any.whl",
        "sampleproject-2.0.0.tar.gz"
    ],
    "public": True
 }],

And the job output:

Here the log shows that artifacts have been stored. Now, as they were marked as public in artifacts step, they are available in the UI:

This is it. In this installment we learn how to clone Git repository, how to run tests using Pytest, how to install dependencies and how to build artifacts and store them in Kraken's storage. More about Kraken CI you can find on https://kraken.ci.

Kraken CI, New Kid on the CI Block

Michal Nowikowski — Fri, 30 Apr 2021 08:41:39 +0000

The main features of the system are:

flexible workflow planning using Starlark/Python
distributed building and testing
various executors: bare metal, Docker, LXD
highly scalable to thousands of executors
sophisticated test results analysis
email and Slack notifications

1. Download Kraken Files for Docker Compose

Download configuration files for Docker Compose from Kraken GitHub releases page.
They are:

kraken-docker-compose-X.Y.yaml - a services configuration file for Docker Compose
kraken-X.Y.env - an example configuration for Kraken services that is used by Docker Compose file

2. Start Kraken Services

In a terminal, in the folder where downloaded files are located, run:

docker-compose --env-file kraken-X.Y.env -f kraken-docker-compose-X.Y.yaml up

Now you should see lots of logs. They will not stop. First, Docker
images are downloaded, and then services are started.

Try to get to the web page on http://localhost:8080.

3. Log In

On the main page, log in as admin using admin/admin credentials.

After logging in, the dashboard page will appear. It shows tiles with
projects and their branches. There is already a prepared project
called Demo.

But first, a built-in agent that will be executing jobs needs to be
authorized.

4. Authorize Agent

Go to the menu Agents -> Discovered. The Discovered Agents page shows
agents that try to connect to the Kraken server but were not yet
authorized by an admin.

Select the checkbox and click the Authorized button.

Now we can go back to the main page (the dashboard) and explore
Demo project. Click on the Kraken logo in the top left corner.

5. The First Project and its Branch

Click the cog icon on the right side of Project Demo. It leads to
the project settings page.

On the project page, there is a list of branches. Let's go to the
already prepared Master branch:

On the branch page, there is a defined workflow that consists of stages.

On the screen, on the left, there is visible one stage, Tests. The
right side presents details of this stage. The black box shows the
critical part of the stage: it is a workflow schema that is written in
Starlark/Python.

Currently, the stage's workflow pulls a sample project from
https://github.com/Kraken-CI/sample-project-python.git and then run
pytest tests.

6. Trigger the First Flow

Click the Run CI Flow button to trigger the first flow of the branch
workflow, and then confirm it by clicking the Submit button.

The next page shows triggered flow details.

On the left side, there is a list of runs of stages (currently there is
only one run: Tests), and on the right, there are details of this
stage run. All is green, so it means that all completed successfully.

7. Results of The Run

Click the Tests run link to get to results of the run.

The first tab, Jobs, on the run results page, shows a list of executed
jobs. Our job was pretty quick, so we see the final results. If a
job takes more time, then here will be presented its live status and
logs.

The next tab, Test Results, presents a table with test cases and their
status (passed, failed, etc.).

We may conclude the quick start guide here. We encourage you to try
play with the workflow schema and check what can be accomplished more.