Forem: Giannis Papadakis

Mutants, Mutants everywhere! Have we "J"est the Mutants?

Giannis Papadakis — Fri, 24 Mar 2023 11:35:28 +0000

In previous post we have covered how to measure code coverage for our React Apps. In this one we will go one step further and review if our tests are meant to test what they should!

Relying on code coverage for assuring test efficiency is not always the best approach when it comes to the testing approach for your shift-left testing strategy.

Mutation testing is the evolution to better and stronger development practises. We need to use it to reassure that our tests are testing something…(100% code coverage from unit tests does not guarantee that we have also strong unit tests!!)

Some history behind it...

Mutation testing was originally proposed by Richard Lipton as a student in 1971, and first developed and published by DeMillo, Lipton and Sayward. The first implementation of a mutation testing tool was by Timothy Budd as part of his PhD work (titled Mutation Analysis) in 1980 from Yale University.

And what exactly is mutation testing???

Bugs, or mutants, are automatically inserted into your production code. Your tests are run for each mutant. If your tests fail then the mutant is killed. If your tests passed, the mutant survived. The higher the percentage of mutants killed, the more effective your tests are.

It's that simple…

Stryker wants to kill all mutants

Stryker will only mutate your source code, making sure there are no false positives. In the beginning of our journey found out that stryker takes significant time to analyse our projects. We aimed to use it for our react microfrontends thus used stryker-js and jest-runner for the RTL tests. One of our first findings was that the time it takes to finish mutation testing was significant HIGH and lets see the reason for this:

Faults are introduced into the source code by creating multiple versions of the code, each version is called a mutant. Each mutant contains a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.
Test cases are applied to the original program and also the mutant program.
Compare the results of the original and mutant program.
If the original program and mutant programs generate the different output, then that the mutant is killed by the test case. Hence the test case is good enough to detect the change between the original and the mutant program.
If the original program and mutant programs generate the same output, mutant is kept alive. In such cases, more effective test cases need to be created that kill all mutants.

The costly part is the inner loop of course, where the tool needs to build and test each mutant. For example, for stryker-js generates around 1600 mutants, and a test run takes around 10 seconds. This give a total of roughly 4 (four) hours. Run time can be significantly improved by using test coverage detail so the engine only run tests that may be impacted by the mutation. But it implies a tight collaboration between the test runner, the coverage tool and the mutation testing tool. Of course improvements have been made and we will cover the needed changes from the team to maximise performance of the engine.

So Stryker has to generate mutants, but this raises two conflicting goals:

On one hand, you want to inject a lot of mutants to really exert your tests.
But on the other hand, you need to have an acceptable running time, hence a reasonable number of test runs (= mutants). This is also demanding if you decide to incorporate mutation testing within your CI pipelines…

The example

Lets start by a simple example first to understand how mutation testing really works.

function isdoubledigit(digit){
   return digit.value >= 10;
}

Stryker will find the return statement and decide to change it in several ways:

/* 1 */ return digit.value > 10;
/* 2 */ return digit.value < 10;
/* 3 */ return false;
/* 4 */ return true;

We call those modifications mutants. After the mutants have been found, they are applied one by one, and your tests are executed against them. If at least one of your tests fail, we say the mutant is killed. That's what we want! If no test fails, it survived. The better your tests, the fewer mutants survive. Stryker will output the results in various different formats. One of the easiest to read is the clear text reporter:

Instructions

Lets setup the project by installing dependencies:

yarn add @stryker-mutaror/core -D
yarn add @stryker-mutator/jest-runner -D
yarn add @stryker-mutator/typescript-checker -D

With typescript checker we enable type checking on mutants.
👽 Type check each mutant. Invalid mutants will be marked as CompileError in your Stryker report.
🧒 Easy to setup, only your tsconfig.json file is needed.
🔢 Type check is done in-memory, no side effects on disk.
🎁 Support for both single typescript projects as well as projects with project references (--build mode).

Create your stryker.conf.js in root dir (Or you can generate by running stryker init). We will explain some configurations you can pass to improve performance.

To run stryker scan just execute:

 yarn stryker run

Let's review the example of configurations to showcase the improvements you can make to the overall execution time.

concurrency
Set the concurrency of workers. This defaults to n-1 where n is the number of logical CPU cores available on your machine, unless n <= 4, in that case it uses n. This is a sane default for most use cases.

coverageAnalysis
all: Stryker will automatically collect code coverage results during the initial test run phase
perTest: Nice! We're already saving time by analysing a simple code coverage result. But if we take a closer look, we see that we can save even more time. Stryker will automatically collect code coverage results per test during the initial test run phase. Next, it will select only those tests that actually cover a mutant to run for that mutant. This might seem like a small improvements, but in big projects with hundreds of tests, it quickly adds up to minutes.

mutate
With mutate you configure the subset of files or just one specific file to be mutated. These should be your production code files, and definitely not your test files. Excluding test files or static files see that execution time improved even further!

If you use perTest you might want to exclude static mutants as well (A static mutant is a mutant that is executed once on startup instead of when the tests are running) to improve the performance even better.

Lets see the results while running from console log:

Nice!! 👌 Now lets see how different the statement coverage from mutation coverage to show the true value of it. We picked up one package which the line coverage is around 95% but the mutation coverage at the same time drops to 50%.

So lets summarise some important pros and cons for our new type of testing below.

Pros

Identifying the number of mutants that have survived vs killed is a more reliable metric than simply using line coverage. It actually ensures your unit tests are testing what they should be.
It catches many small programming errors, as well as holes in unit tests that would otherwise go unnoticed.

Cons

Running a mutation testing framework against an entire complex project is computationally very expensive, requires a lot of processing power to complete.
Runs can take anywhere up to several hours, making them unsuitable within a fast release process. Of course, you can run a framework overnight and check the report later.
You need to build metrics around surviving mutants to ensure they are dropping in numbers over time

Conclusion
We do not want to compare code coverage with mutation coverage just point out that you may need a way to measure the effectiveness of your unit tests. So we can empower more testing capabilities within our teams. I would certainly urge you to give it a try and see what is missing from your testing strategies!!!

Code Coverage with React, Vite, RTL & Cypress

Giannis Papadakis — Fri, 17 Mar 2023 09:22:18 +0000

My newest post involves one of our recent projects to move our react microfrontend to use Vite. I will summarise in this post how we managed to collect code coverage statistics from RTL and Cypress tools to have a combined coverage level for our development process. But before jumping to technical details let's review the background story.

Initially using Webpack and decided to move to Vite for reasons that you may already know like:

It aims to provide a faster and more efficient development experience for developers by using the native ES modules feature in the browser
Ability to handle large JavaScript projects with ease
Excellent documentation

The above are just some few key features of Vite that makes it stand out from other solutions.

We already know what is Cypress?

With Cypress, you can easily create tests for your modern web applications, debug them visually, and automatically run them in your continuous integration builds. It can easily be integrated in your react projects and run both component and integration tests at ease as part of your development pipeline.

But what about your component tests?

If the team is more flexible you can use Cypress even to test your react components but most of the teams are more aware of RTL(react testing library) so in our projects as well we stick to use this to drive our component tests.

And what about code coverage?

When it comes to code coverage we picked Instanbul.
Istanbul instruments your ES5 and ES2015+ JavaScript code with line counters, so that you can track how well your unit-tests exercise your codebase. The nyc command-line-client for Istanbul works well with most JavaScript testing frameworks: tap, mocha, AVA, etc.

Now that we had some few introductions lets deep dive into the setup needed to start measuring code coverage on your react project.

First install the needed tool to instrument your react app as Vite plugin yarn add vite-plugin-istanbul -D.

After we installed the package we need to change the vite.config.ts file accordingly

From the configuration above you will see that we used instanbul for both cypress e2e & rtl tests.

But we will not see all the files because Istanbul by default uses coverage only for those files for which there are tests, in order to configure Istanbul reports more flexibly, add the following package:

yarn add @istanbuljs/nyc-config-typescript -D

Now create the .nycrc.json to store settings for cypress tests.

Ok great we can now generate reports for each tests (RTL & Cypress) but it would be extremely useful to merge both under one single report.

Conclusion
That’s all, thanks for reading, in this article we figured out how to add code coverage support to our application using React and Vite for both unit and e2e tests.

Integrate K6 with InfluxData

Giannis Papadakis — Tue, 17 Jan 2023 10:39:17 +0000

In this post we will describe the process of integrating k6.io with InfluxData (InfluxDB cloud). We recently integrated load tests with k6 in our development process in GWI and our journey just begun.

InfluxDB v1 is already supported from k6 out of the box to persist metrics but support on v2 is not yet fully covered. Let's start by reviewing how to deploy a local instance of v1 and integrate with k6 for the sake of introduction.

InfluxDB V1

First we need to create our grafana-datasource.yaml file to provision Grafana, with the following configuration

apiVersion: 1
datasources:
  - name: myinfluxdb
    type: influxdb
    access: proxy
    database: k6
    url: http://influxdb:8086
    isDefault: true

Let's create also the docker-compose file that will do the following:

Setup InfluxDB v1
Setup Grafana with predefined dashboards
Run k6 tests and direct output to InfluxDB

networks:
  k6:
  grafana:
services:
  influxdb:
    image: influxdb:1.8 # Version 2.x introduces some breaking compatibility changes. K6 support for it comes via an extension
    networks:
      - k6
      - grafana
    ports:
      - "8086:8086"
    environment:
      - INFLUXDB_DB=k6

  grafana:
    image: grafana/grafana:latest
    networks:
      - grafana
    ports:
      - "3000:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_BASIC_ENABLED=false
    volumes:
      - ./grafana-datasource.yaml:/etc/grafana/provisioning/datasources/datasource.yaml
  k6:
    image: grafana/k6:latest
    networks:
      - k6
    ports:
      - "6565:6565"
    environment:
      - K6_OUT=influxdb=http://influxdb:8086/k6
    volumes:
      - ./dist:/scripts

Running our tests we can easily review and analyze through the Grafana Dashboards

InfluxData

InfluxData is the cloud version of InfluxDB and it is based on V2. Main difference is that is organized in buckets and we can query data based on Flux Query.

The current Grafana dashboards are mainly developed to support V1 so how can we support backwards compatibility without creating our own dashboards?

First thing first lets review how the docker-compose file is changes to persist data to InfluxData:

networks:
  k6:
  grafana:
services:
  grafana:
    image: grafana/grafana:latest
    networks:
      - grafana
    ports:
      - "3000:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_BASIC_ENABLED=false
    volumes:
      - ./grafana-datasource.yaml:/etc/grafana/provisioning/datasources/datasource.yaml
  k6:
    build: .
    networks:
      - k6
    ports:
      - "6565:6565"
    environment:
      - K6_OUT=xk6-influxdb=<influxdb_url>
      - K6_INFLUXDB_ORGANIZATION=<influxdb_org_id>
      - K6_INFLUXDB_BUCKET=Platform_Performance_Tests
      - K6_INFLUXDB_INSECURE=true
        # NOTE: This is an Admin token, it's not suggested to use this configuration in production.
        # Instead, use a Token with restricted privileges.
      - K6_INFLUXDB_TOKEN=<influxdb_token>
    volumes:
      - ./dist:/scripts

The change appararently is the output and here we use the xk6 binary xk6-influxdb
Provide proper environment variables retrieved from your InfluxData configuration.

Now we can actually persist our data to InfluxData lets see how to use Grafana Dashboards from V1 with backwards compatibility.

The InfluxDB 1.x data model includes databases and retention policies. InfluxDB Cloud replaces databases and retention policies with buckets. To support InfluxDB 1.x query and write patterns in InfluxDB Cloud, databases and retention policies are mapped to buckets using the database and retention policy (DBRP) mapping service.

Our dashboards are using InfluxQL (v1) to query data so create a datasource with InfluxQL

Now the only thing remain is to add custom HTTP header Authorization and provide as value the InfluxData token to make appropriate connection. So we are good to go and use the predefined dashboards from grafana site

Conclusion

If someone is experienced enough using Flux Query you do not need to worry about backwards compatibility from v2 to v1 API. But yet again i wanted to share my experience trying to move to InfluxData and how we managed to migrate our Grafana Dashboards without any extra handling!

Web Performance with SiteSpeed.io

Giannis Papadakis — Fri, 24 Jul 2020 07:06:06 +0000

In this post we will cover basic concepts for Web Performance and with what frameworks we can measure it in modern web applications.

Why should we care for Web Performance??

Two words you often hear together are mobile and site speed. And that’s not without reason because these two go hand in hand. Mobile-friendliness and site speed are some of the most pressing matters we have to deal with.

Measuring page speed has always been something of a dark art. The site speed tools we use today are fairly adequate, but with the new Web Vitals metrics Google is trying to come at it from a different, more realistic angle, taking page experience into account.

The first thing to understand is that there is no single metric or measurement for ‘speed’. There’s no simple number which you can use to measure how quickly your pages load.

Think about what happens when you load a website. There are lots of different stages and many different parts which can be measured. If the network connection is slow, but the images load quickly, how ‘fast’ is the site? What about the other way around?

Even if you try to simplify all of this to something like “the time it takes until it’s completely loaded“, it’s still tricky to give that a useful number.

For example, a page which takes longer to ‘finish loading’ may provide a functional ‘lightweight’ version while the full page is still downloading in the background. Is that ‘faster’ or ‘slower’ than a website which loads faster, but which I can’t use until it’s finished loading?

Browser Stuff

This stage is where the page needs to be constructed, laid out, colored in, and displayed. The way in which images load, in which JavaScript and CSS are processed, and every individual HTML tag on your page affects how quickly things load.

We can monitor some of this from the ‘outside-in’ with tools which scan the website and measure how it loads. We recommend using multiple tools, as they measure things differently, and are useful for different assessments. For example:

How to diagnose speed issues through the waterfall charts:

1.Look for red text

If you have 400 or 500 error response codes for resources in your waterfall chart, you may see the name of the resource marked in red text. This indicates an error retrieving that resource. The below example shows a 500 Internal Server Error on a waterfall chart, but you could see any individual request fail within the makeup of a page.

2.Look for long bars

In the following example, the long bar represents an exceptionally long time for a DNS lookup to find a custom font on a site.

3.Look for big gaps between requests

As demonstrated in the following example, these gaps represent times when no requests happened, such as when JavaScript was executing or the browser was processing.

Universal Metrics

Despite all of these moving parts, there are a few universal metrics which make sense for all sites to measure, and optimize for. These are:

Time until first byte, which is how long it takes until the server responds with some information. Even if your front-end is blazing fast, this will hold you up. Measure with Query Monitor or NewRelic.
Time until first contentful (and meaningful) paint, which is how long it takes for key visual content (e.g., a hero image or a page heading) to appear on the screen. Measure with Lighthouse for Chrome.
Time until interactive, which is how long it takes for the experience to be visible, and react to my input. Measure with Lighthouse for Chrome.

These are much more sophisticated metrics than “how long did it take to load”, and, perhaps more importantly, have a user-centric focus. Improving these metrics should correlate directly with user satisfaction, which is super-important for SEO

Measuring Web Performance with SiteSpeed.io

When it comes to more modern tools SiteSpeed.io covers multiple metrics and has the advantage to integrate into a more modern DevOps Pipeline.

Sitespeed.io is a set of Open Source tools that makes it easy to monitor and measure the performance of your web site.

Built-in Simulation Features

Browser Support
Mobile
Simulation Real Device (Android for now)
Bandwidth Simulation
Continuous Integration
Selenium Integration
Crawling

Third Party Tools Integration

Slack Integration
Google Page Speed Insight Integration
WebpageTest Integration
Various Plugins
Custom Metrics

Reports

Grafana Dashboards
Coach
Unitary Report
Historical Dashboard

SiteSpeed.io in CI/CD

To run SiteSpeed.io its pretty simple especially if you use the Docker Image

docker run --shm-size=1g --userns=host --rm -v ${WORKSPACE}:/sitespeed.io sitespeedio/sitespeed.io:13.3.2-plus1 --outputFolder output/report --axe.enable true  --video false --influxdb.host ... --influxdb.port .... --influxdb.database ... --influxdb.username ... --influxdb.password ... http://www.google.com --slack.hookUrl ...

In the above example we have various arguments passing summarized:

--outputFolder: The folder relative to workspace to store the results
--axe.enable: Enable support for accessibility scores
--video: Enable or disable video support
--influxdb.host: InfluxDB host to store metrics as time series data
--influxdb.port: InfluxDB port
--influxdb.database: InfluxDB database name to store the metrics
--slack.hookUrl: Enable WebHooks to Slack channels and update the channel with performance output

Of course this is just an example if you want to add more capabilities in your docker image command visit SiteSpeed Configurations

The output after scanning will produce multiple html reports combined in one dashboard providing scores and individual metrics per pages

When you have to deal with scanning the application when registration forms or login forms are providing more data for the user then using only the docker image to scan will not be sufficient. No problem ... Selenium is our tool.

You can write a Selenium async script(EventCloud2.js) and pass it to docker image:

module.exports = async function(context, command) {
       // Use case #1: Open EventCloud Page
   await command.measure.start('http://eventcloud.aspnetboilerplate.com/', 'Load EventCloud');  
    try
   {
       // Use case #2: Open Register Page
       await command.navigate('http://eventcloud.aspnetboilerplate.com/account/login');
       await command.measure.start('Open_Register');
       await command.click.byClassName('btn btn-block bg-deep-purple waves-effect');
       context.log.info('Register page opened..');
       await command.measure.stop();

       // Use case #3: Login Page.
       await command.navigate('http://eventcloud.aspnetboilerplate.com/account/login');
       await command.wait.byTime(5000);
       await command.addText.byXpath('john', '//input[@name="userNameOrEmailAddress"]');
       await command.addText.byXpath('123qwe', '//input[@name="password"]');

       await command.measure.start('Login');
       await command.click.byIdAndWait('LoginButton');
       context.log.info('User logged In..');

       return command.measure.stop();
   }
   catch(e) {}
};

Then just execute the following command:

docker run --shm-size=1g --userns=host --rm -v ${WORKSPACE}:/sitespeed.io sitespeedio/sitespeed.io:13.3.2-plus1 --outputFolder output/report --axe.enable true  --video false --influxdb.host ... --influxdb.port .... --influxdb.database ... --influxdb.username ... --influxdb.password ... EventCloud2.js --slack.hookUrl ...

Grafana Integration

In order to display the metrics, we need to create the SiteSpeed dashboard. For this specific task, we are going to import a JSON script by following these steps:

• Click the “+” icon on the left menu > “Import.”

• In the text box, paste the JSON script copied from this link.

• Click the “Load” button.

• In the import options, set a name for your dashboard, the folder where you want to have it, and the InfluxDB data source created in step 5.

• Click the “Import” button.

Now you can review above widgets in Grafana containing most of the information in html reports:

Slack Integration

You can enable WebHook into a public/private Slack Channel to send information for running scans to the team as well.

Create a new Slack app in the workspace where you want to post messages.
From the Features page, toggle Activate Incoming Webhooks on.
Click Add New Webhook to Workspace.
Pick a channel that the app will post to, then click Authorize.
Use your Incoming Webhook URL to post a message to Slack
Copy the WebHook URL to use it in argument --slack.hookUrl

Then you will see results summarized and posted to Slack:

Wrapping this into a process

Use an ‘outside-in’ tool, like SiteSpeed or Lighthouse to generate a waterfall diagram of how the website loads.
Identify bottlenecks with servers and the back end. Look for slow connection times, slow SSL handshakes, and slow DNS lookups. Use a plugin like Query Monitor, or a service like NewRelic to diagnose what’s holding things up. Make server, hardware, software and script changes.
Identify bottlenecks with the front end. Look for slow loading and processing times on images, scripts and stylesheets. Use a tool like SiteSpeed.io we described previously to measure also more metrics, like time until first meaningful paint and time until interactive.

Web Security Testing with OWASP ZAP and Selenium

Giannis Papadakis — Wed, 06 May 2020 13:03:52 +0000

Have you ever wondered how we can actually find security vulnerabilities in Web Applications? There are guidelines from global security organizations that can be followed from Security Experts on how to efficiently perform penetration and security tests in your application. To review the top 10 vulnerabilities refer to OWASP Top 10 Risks

Introduction to Security Testing

There are multiple scanners in the software community commercial or open-source that gives the ability to penetration testers and security engineers to scan their application for known vulnerabilities.

Now most of the scanners are having CI/CD support and works well side by side with Selenium which is the tool that simulates user actions in our browsers.

OWASP Zed Attack Proxy (ZAP)

The OWASP Zed Attack Proxy (ZAP) is one of the world’s most popular free security tools and is actively maintained by hundreds of international volunteers. It can help you automatically find security vulnerabilities in your web applications while you are developing and testing your
applications. Its also a great tool for experienced pen testers to use for manual security testing.

Objective

To use OWASP ZAP, to detect web application vulnerabilities in a CI/CD pipeline

Problem

Web applications have Basic Authentication, User Logins and Form Validation which stops Scanner in its tracks

Solution

Use Selenium test scripts to drive ZAP. A project may include already selenium scripts for functional testing. Active scans actively modify the recorded requests and responses to determine further vulnerabilities

CI/CD Setup

Let's create a CI pipeline that will start ZAP in headless mode, run our functional tests that will perform two types of scan (active/passive), store results of scanning alerts in HTML Reports and tear down the server.

CI/CD Steps:

Start ZAP
Run Selenium Scripts (Passive Scan)
Wait for Passive scan to complete
Start Active Scan
Wait for Active scan to complete
Retrieve alerts and report

OWASP ZAP Installation

OWASP ZAP can be installed with multiple ways but we prefer to use Docker which is the simplest way to bring up the server.

 stage('OWASP ZAP setup'){
    sh "docker pull owasp/zap2docker-stable"
    sh "docker run --rm -d -u zap --name zap -p 4449:4449 -i 
        owasp/zap2docker-stable 
        zap.sh # A start up script provided by ZAP
        -daemon # Start in a headless configuration
        -host 0.0.0.0 # The ZAP host
        -port 4449 # The ZAP port 
        -config api.addrs.addr.name=.*
        -config api.addrs.addr.regex=true # Allow any source IP to connect 
        -config api.key=testypon" # Api key to be used 
    }

Now that we brought up ZAP in headless mode navigating to localhost in port 4449 you will be able to see:

WebDriver Integration

Lets see how we can integrate ZAP with our WebDriver instance that will drive the user interaction with our application:

DesiredCapabilities caps = new DesiredCapabilities();
Proxy proxy = new Proxy();
proxy.setProxyType(Proxy.ProxyType.PAC);
StringBuilder strBuilder = new StringBuilder();
strBuilder
.append("http://localhost:4449")
.append("/proxy.pac?apikey=")
.append(zapApiKey);
proxy.setProxyAutoconfigUrl(strBuilder.toString());
caps.setCapability(CapabilityType.PROXY, proxy);

Now pass the capabilities in your WebDriver Object as usual to intercept traffic from ZAP.

If the target web application has security response headers in place, specifically Strict-Transport-Security the WebDriver should be configured as follows:

caps.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
caps.setCapability(CapabilityType.ACCEPT_INSECURE_CERTS, true);

Passive Scan

Passive scans record the requests and responses sent to the web application and creates alerts for detected vulnerabilities. Also they are triggered whenever we access the application from WebDriver.

Active scan

Active scans actively modify the recorded requests and responses to determine further vulnerabilities for the application.

ZAP API in Selenium

For the reasons of simplicity we created a Driver(Wrapper to the actual ZAP API) called ZAPDriver to map to all needed API calls from our selenium scripts and will drive the execution of the scans. Lets see the most needed fuction for the scans:

First add your dependencies in the test project:

<dependency>
 <groupId>org.zaproxy</groupId>
 <artifactId>zap-clientapi</artifactId>
 <version>${zapapi.version}</version>
</dependency>

Now lets create our ZAPDriver to start interacting with the API functions:

@Service
@Profile("Security")
public class ZapDriver implements Spider, ScanningProxy, ContextModifier, Authentication {

    @Value("${zap.enabled:false}")
    private boolean zap;

    @Value("${zap.host:}")
    private String zapHost;

    @Value("${zap.base.url}")
    private String zapBaseUrl;

    @Value("${zap.port:0000}")
    private int zapPort;

    @Value("${zap.api.key}")
    String zapApiKey;

    private static final String MINIMUM_ZAP_VERSION = "2.6"; // Weekly builds are also allowed.

    private ClientApi clientApi;

    @PostConstruct
    public void initializeScanner() {
        this.clientApi = new ClientApi(zapHost, zapPort, zapApiKey);
        this.secData = new SecurityData();
        validateMinimumRequiredZapVersion();
        setAttackMode();
    }

In the PostContruct phase of the Object we initialize an instance of the ClientAPI to perform the need HTTP calls to ZAP from within our tests

Step 1: Enable Scanner

Now if you enable Passive or Active scanners you can do that with different ways. You can find the scanner policy name from the API and use the id to enable specific scanners for example (SQL Injection or Cross Site Scripting) but for the sake of simplicity we can enable all scanners and get a unified report with different categories of alerts:

     @Override
    public void enableAllScanners() throws ProxyException {
        try {
            clientApi.pscan.setEnabled("true");
            ApiResponse response = clientApi.ascan.enableAllScanners(null);
            log.trace(String.format("ZAP OK response for api call %s!!!", response.getName()));
        } catch (ClientApiException e) {
            throw new ProxyException(e);
        }
    }

The above will enable all active scanners and you have also another call if you want to enable all passive scanners as well

Step 2: Spidering

Spider is an Internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing (web spidering).

  @Override
    public void spider(String url) {
        try {
            ApiResponse response = clientApi.spider.scan(url, null, null, null, null);
            log.trace(String.format("ZAP OK response for api call %s!!!", response.getName()));
        } catch (ClientApiException e) {
            log.error("Exception trying to spider " + e.getDetail());
        }
    }
 @Override
    public void excludeFromSpider(String regex) {
        try {
            ApiResponse response = clientApi.spider.excludeFromScan(regex);
            log.trace(String.format("ZAP OK response for api call %s!!!", response.getName()));
        } catch (ClientApiException e) {
            throw new ProxyException(e);
        }
    }

You can exclude also URLs from third party providers that you do not want to spider in order to speed up the crawling process and produce valid alerts for your application.

Step 3: Scanning

After spidering your application you need to scan for gathering our alerts based on the enabled policy of the scanners previously

 @Override
    public void scan(String url) throws ProxyException {
        try {
            ApiResponse response = clientApi.ascan.scan(url, "true", "false", null, null, null);
            log.trace(String.format("ZAP OK response for api call %s!!!", response.getName()));
        } catch (ClientApiException e) {
            throw new ProxyException(e);
        }
    }

Great!!!

Tip: Be careful that the times of the scan are different per application. You need to adjust the scan/spider timeout per your need

Step 4: Reporting

After scan is complete we need to create a report for the alerts and store it on our CI server.

byte[] htmlReport = scanner.getHtmlReport()
Path pathToFile = Paths.get(path)
Files.createDirectories(pathToFile.getParent())
Files.write(pathToFile, htmlReport)
allureService.html(pathToFile.toFile(),"OWASP ZAP Report")

 @Override
    public byte[] getHtmlReport() throws ProxyException {
        try {
            return clientApi.core.htmlreport();
        } catch (ClientApiException e) {
            throw new ProxyException(e);
        }
    }

Now the report for example will summarize all alerts with description and possible solutions provided from OWASP ZAP Organization:

Conclusion

SecDevOps is the philosophy of integrating security practices within the DevOps process. SecDevOps involves creating a 'Security as Code' culture with ongoing, flexible collaboration between release engineers and security teams. There are plenty tools and platforms that can be used in the release cycles of your application and we reviewed how to use the most known open-source tool.

Visual Automation with Applitools

Giannis Papadakis — Fri, 24 Apr 2020 09:58:50 +0000

The purpose of this article is to review how we can integrate Applitools platform to add AI/ML in our existing Web UI Functional Tests.

Applitools Introduction

Applitools is "yet" another tool that you can use in conjunction with Selenium to handle the complicated visual types of testing you need to get done, and leave the more functionality-type testing to the tools that where designed to handle that, like Selenium.

Applitools provide an easy to use platform to review visual differences between screens that can handle different configurations we will examine later on (e.g view port, device types etc.). It provides integration with multiple test frameworks and programming languages in order for any team to be able to adjust to their needs.

Applitools Integration with Test Frameworks

There are a bunch of SDKs for most of the programming languages that allow you to take images and upload them for visual validation in Applitools. We are going to review on how to start with the integration in a Java based Test Project.

Dependencies and SDKs

In your maven dependency management section add the following dependencies:

 <!-- START Applitools Eyes dependencies -->
   <dependency>
       <groupId>com.applitools</groupId>
       <artifactId>eyes-selenium-java3</artifactId>
       <version>${eyes-java3.version}</version>
   </dependency>
   <dependency>
       <groupId>com.applitools</groupId>
       <artifactId>eyes-images-java3</artifactId>
       <version>${eyes-java3.version}</version>
    </dependency>
 <!-- END Applitools Eyes dependencies -->

Before any selenium test starts execution we need to initialize Eyes and EyesRunner objects in order to take images within our tests.

An example of initializing the runner:

 EyesRunner runner;
        if (visualGrid) {
            LOG.info("Applitools: Running on Visual Grid");
            runner = new VisualGridRunner(concurrency);
        } else {
            LOG.info("Applitools: Running on Classic Runner");
            runner = new ClassicRunner();
        }
        Configuration config = new Configuration();
        config.addBrowser(1920, 1080, BrowserType.CHROME)
                .addBrowser(1920, 1080, BrowserType.FIREFOX)
                .addDeviceEmulation(DeviceName.iPhone_X, ScreenOrientation.PORTRAIT)
                .setApiKey(apiKey)
                .setBatch(batch)
                .setMatchLevel(!matchLevel.isEmpty() ? MatchLevel.valueOf(matchLevel) : MatchLevel.LAYOUT);

Lets review the arguments of the above example. First we need to set the type of Runner and currently Applitools support two flavors:

Classic Runner: Applitools will process the images taken in your local executed browser/device
Visual Grid Runner: Applitools give you the ability to send images and process them in different type of configurations (browsers/devices) scaling your tests on different platforms covering more visual checkpoints Next we have the API Key which is provided by an Admin user of the platform. Batch can be used to set the batch name that will be used to store the tests for this runner and finally the Match Level which is the type of validations that will be used to compare the checkpoints.

Applitools Eyes can test the UI in 4 different comparison levels:

Exact (MatchLevel.EXACT) - pixel-to-pixel comparison, not recommended.

Strict (MatchLevel.STRICT) - Strict compares everything including content (text), fonts, layout, colors and position of each of the elements. Strict knows to ignore rendering changes that are not visible to the human. Strict is the recommended match level when running regression tests on the same browser/OS (strict is not designed to do cross browser comparison).

Content (MatchLevel.CONTENT) - Content works in a similar way to Strict except for the fact that it ignores colors.

Layout (MatchLevel.LAYOUT) - Layout, as its name implies, compares the layouts (i.e. structure) of the baseline and actual images. It validates the alignment and relative position of all elements on the page, such as: buttons, menus, text areas, paragraphs, images, and columns. It ignores the content, colour and other style changes between the pages.

Now lets see how to initialize the Eyes instance

Eyes eyes = new Eyes(runner);
eyes.setConfiguration(config);
eyes.setForceFullPageScreenshot(true);
eyes.setStitchMode(StitchMode.CSS);
eyes.setWaitBeforeScreenshots(2000);
eyes.setLogHandler(new FileLogger("target/eyes.log", true, true));

Now that we initialized our eyes before our test we can actually navigate to a page with selenium and take a baseline image from the page and send it over to Applitools.

log.info("Start Visual Check From Applitools");
int matchTimeout = 10000;// in milli seconds
if (!eyes.getIsOpen()) {
    eyes.open(driver, "VisualTest","TestName");
}
    eyes.checkWindow(matchTimeout, "ScreenshotName");
}

With the commands above we create a new test and send an image of the page into Applitools for visual comparison.

Getting Results from Applitools

First we need to close eyes instance after a test is finished execution with

boolean testFailed = result.getStatus() == ITestResult.FAILURE;
if (!testFailed) {
      eyes.closeAsync();
     } else {
      eyes.abortAsync();
     }
}

This terminates the sequence of checkpoints, and then waits asynchronously for the test results.

We can retrieve the results now by calling:

TestResultsSummary allTestResults = runner.getAllTestResults(false);
for (TestResultContainer result1 : allTestResults) {
    Throwable ex = result1.getException();
    TestResults result = result1.getTestResults();
    if (result == null) {
    LOG.info("No test results information available\n");
    } else {
String resultReport = String.format("URL = %s, AppName = %s, testname = %s, Browser = %s,OS = %s, viewport = %dx%d, matched = %d,mismatched = %d, missing = %d,aborted = %s\n", result.getUrl(),
 result.getAppName(),
 result.getName(),
 result.getHostApp(),
 result.getHostOS(),
 result.getHostDisplaySize().getWidth(),
 result.getHostDisplaySize().getHeight(),
 result.getMatches(),
 result.getMismatches(),
 result.getMissing(),
 (result.isAborted() ? "aborted" : "no"));
 LOG.info("Applitools Results: " + resultReport);
}

So with this way we can get results and post them to our test reporting platforms.

Applitools CI/CD Integration

In order to view test results directly in your Jenkins CI server you need first to install Applitools Jenkins plugin.

Then we need to add the following section in your jenkins pipeline stage for running the visual tests:

node {
     stage('Visual Tests') { 
         Applitools() {
               sh 'mvn clean test'
         }
     }
}

After that the results are available in an embedded iframe in Jenkins:

Integrate ReportPortal.io in Java Projects

Giannis Papadakis — Thu, 16 Apr 2020 10:45:27 +0000

In this post we will describe how someone can integrate ReportPortal platform for an existing java test project in order to leverage reporting of unit/integration tests. ReportPortal is an AI-powered Test Automation Dashboard that you can use for visualizing your test suites across multiple components and products and provides a unique real time experience.

Setup ReportPortal.io

In order to setup ReportPortal we need to have preinstalled docker-compose locally.
Download docker-compose.yml from ReportPortal.io by executing the following command:

curl https://raw.githubusercontent.com/reportportal/reportportal/master/docker-compose.yml -o docker-compose.yml

Start then the platform by executing the following docker-compose command:

docker-compose -p reportportal up -d --force-recreate

Now if you navigate to localhost:8080 you will see the Login Screen where you can login with following default credentials superadmin\erebus

Integrate with Test Frameworks

Now that we have the platform up and running lets review an example of a test project and how to integrate with RP to forward test results. For the purpose of this i will be using TestNG but of course there are multiple adaptors that someone can review in the official RP Documentation.

Adding Dependencies

In order to start we need to add the dependencies for reportportal in our test project. Locate your maven pom.xml and add the following:



 <dependency>
   <groupId>com.epam.reportportal</groupId>
   <artifactId>logger-java-logback</artifactId>
   <version>3.0.0</version>
</dependency>

<dependency>
  <groupId>com.epam.reportportal</groupId>
  <artifactId>agent-java-testng</artifactId>
  <version>3.0.0</version>
</dependency>

Also we need to add a private repository storing the external dependencies:

<repositories>
     <repository>
        <snapshots>
          <enabled>false</enabled>
        </snapshots>
        <id>bintray-epam-reportportal</id>
        <name>bintray</name>
        <url>http://dl.bintray.com/epam/reportportal</url>
     </repository>
</repositories>

Now in order to send results to your ReportPortal instance first we need to create a project and get following parameters stored in a file called reportportal.properties that should be stored in your classpath under src/test/resources.

rp.endpoint = http://localhost:8080
rp.uuid = 2c54960d-cb95-4842-9c9b-5e72e43d1979
rp.launch = Test_Launch
rp.project = superadmin_personal

Posting Data to ReportPortal

Now that we added needed dependencies we need to enable the adapter and also review how we can send custom data to RP like logs,attachments etc.

There are two ways to add the adaptor in TestNG:

Annotation Listeners:

@Listeners({ReportPortalTestNGListener.class})
public class FailedParameterizedTest {
…..
}

TestNG XML:

<suite>

  <listeners>
    <listener class-name="com.example.MyListener" />
    <listener class-name="com.epam.reportportal.testng.ReportPortalTestNGListener" />
  </listeners>
.....

Now we enabled to start having launch details triggered when our tests are triggered

Logging Framework Support

In our case as logging framework we use logback-classic and slf4j so in order to send our test logs to RP locate logback.xml and add the following:

   <appender name="ReportPortalAppender" class="com.epam.reportportal.logback.appender.ReportPortalAppender">
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%t] %-5level - %msg%n</pattern>
        </encoder>
    </appender>

    <root level="info">
        <appender-ref ref="STDOUT" />
        <appender-ref ref="FILE" />
        <appender-ref ref="ReportPortalAppender" />
    </root>

Now lets see an example of attaching a screenshot for a failed Web Test.
In order to send a file as attachment we will use our logger with the following arguments:

log.info("RP_MESSAGE#FILE#{}#{}", ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE).getAbsoluteFile(), "Screenshot on Failure");

First argument is the File to be attached and the second the message we want to be shown in the report

See an example attached here:

Conclusion

ReportPortal is one example of how AI/ML is transforming software testing and can be easily integrated in your existing test infrastructure
If you want:

Manage all your automation results and reports in one place
Make automation results analysis actionable & collaborative
Establish fast traceability with defect management
Accelerate routine results analysis
Visualize metrics and analytics
Make smarter decisions together

Try it out!!!