Forem: Talank

Best Practices in Regression Testing for Beginners

Talank — Tue, 11 Jan 2022 03:42:29 +0000

Change is a vital process in software maintenance, and during changes, new bugs might occur. Even if you carried out a thousand different test scenarios before releasing the first version of the software, it is not guaranteed that the second version does not require any of those tests to be re-executed. This is because fixing new bugs might cause some older bugs that were fixed in an earlier release to reappear.

So, to ensure that old bugs have not resurfaced, we conduct regression testing. Regression testing is the testing that is done before finalizing any changes in the source code of the software.

There are several levels of tests, for example, unit tests, integration tests, end-to-end tests, etc. And regression tests could comprise any of these tests. It is not about the level of tests or the type of tests, but about the time when the tests are done and the depth of tests. It is about time because the regression test is done before any changes are merged or deployed. As for the depth, the greater the coverage, the better the quality assurance.

Nevertheless, it may become very tedious to manually test all the scenarios from a test plan if the number of test scenarios is very high. A solution to this problem is automated regression testing.

You might have already heard some buzzwords around this topic, such as BDD, TDD, test report, bug tracking, etc. Many tools and technologies can indeed be used to automate testing your software. But, my advice is that you should not get intimidated by these tools and technologies while you are a beginner in regression testing.

Regression testing is mainly about re-executing the tests several times. Sometimes, it might not be possible to execute all the tests, so you might have to categorize the test cases based on their priorities and context as a tester.

If you are in the starting phase of your software testing career, then I can give you the following tips to perform regression testing in the best way possible.

1. Automate as much as possible

source
One of the seven principles of software testing states that “Exhaustive testing is not possible”; however, it is always better to test as much as we can. And automating as many tests as possible makes the process of regression testing a lot faster. So you should try to increase test coverage with more automated tests.

And add varieties of tests. Do not just write unit tests, although they catch more bugs than any other tests. You should write end-to-end tests as well that reflect the user stories.

2. Run tests in some CI servers and run them parallelly

Automated regression tests should be executed before any pull request is merged into a base branch. As the number of tests increases, these tests might take hours to complete execution. So, instead of running the tests each time in a local machine, we run them in the CI servers, such as Travis, Drone, GitHub Actions, etc. Finally, as far as possible, run the tests concurrently so that developers do not have to wait long for the test to be done.

3. Maintain your test because the AUT is changing the behavior

source
Once you automate your test scenarios, you should run the test on each commit and maintain the taste according to the changes made in each commit. If the change is unintentional, then the taste fails, which will notify the developer that the regression test detected a new bug. And if the change is intentional, then the test might fail, in which case software testers should maintain the test scenario to comply with the new behavior of the application under test (AUT).

4. Priorities the tests because running each test becomes time-consuming

After you automate a lot of test cases, you should prioritize those tests so that the most important tests are executed in every commit. For this purpose, you can tag your tests based on the area that it covers. One of the popular tags is “smokeTest”, which is used to indicate that the test is very basic and there is no point in executing other tests if the smoke test fails. If you tag your tests based on the context, you can execute the tests only if the changes are related to that context. It will save a significant amount of time.

5. Don't skip the test that caught some bugs previously

The defect clustering principle of software testing explains that almost 80% of the defects are centered around 20% of the code (or module). So, if a test finds some bug, you should run that test on each regression test because it is highly likely that the same test will catch a similar bug.

Conclusion

In a nutshell, regression testing requires tests to be executed multiple times throughout the SDLC. So, to avoid boredom and increase the efficiency of your work, you should try to automate as many tests as you can. Similarly, classify the tests and set the priorities so that you would know which tests must be executed for change in particular modules.

Visual Regression Test with Nigthwatch VRT

Talank — Wed, 03 Nov 2021 07:40:26 +0000

Firstly, if you don't have any idea about what Visual Regression Test (VRT) is then I would recommend that you read my previous blog Insight to Visual Regression Testing. If you have already read that one or you already have some idea about "What is VRT?", then you are now ready to read this blog. The figure below (By Galaxy Weblinks) also gives a basic idea about VRT and the various tools that can be used for automated VRT.

The definition of nightwatch-vrt is quite clear from its name. It is a VRT tool that works as an extension to nightwatch.js. Using nightwatch-vrt is pretty straightforward. You only need to invoke the screenshotIdenticalToBaseline function to make a comparison of the current screenshot with the respective baseline image. And if you look at the internal logic of this function then you will find that all it does is wait for an element to be present, then capture the screenshot of that element, compare it with the baseline, and finally return the comparison result. And if you look further into the image comparison logic then you will notice that nightwatch-vrt actually uses JIMP (JavaScript Image Manipulation Program) which is built with Javascript with no native dependencies.

Configuration

As I already said that nightwatch-vrt is an extension to the nightwatch.js, you first need to setup nightwatch.js. And if you don't know how to setup nightwatch.js then you can refer to one of my previous blogs Setup Nightwatch and run your first test
. Next, add nightwatch-vrt in the devDependencies of your project. You can do it using the following command using yarn.

yarn add --dev nightwatch-vrt

Now, you need to add/edit the custom_commands_path and custom_assertions_path in your nightwatch configuration file as follows.

custom_commands_path: [
        'node_modules/nightwatch-vrt/commands'
    ],
    custom_assertions_path: [
        'node_modules/nightwatch-vrt/assertions'
    ]

In addition to the above-mentioned setup, you also need to add visual_regression_settings to the nightwatch configuration file's globals section as follows.

globals: {
                visual_regression_settings: {
                    generate_screenshot_path: generateScreenshotFilePath,
                    latest_screenshots_path: 'tests/vrt/latest',
                    baseline_screenshots_path: 'tests/vrt/baseline',
                    diff_screenshots_path: 'tests/vrt/diff',
                    threshold: 0.02,
                    prompt: false,
                    always_save_diff_screenshot: false
                }
            },

And you also need to define the function that you specified in generate_screenshot_path. For example, in the above configuration, I have used the function generateScreenshotFilePath which I defined as follows.

function generateScreenshotFilePath(nightwatchClient, basePath, imagePath) {
    return path.join(process.cwd(), basePath, imagePath)
}

We pass a function to generate_screenshot_path because the VRT screenshot path generator option accepts a function that returns a string containing the full path based on the test properties. We can not provide a constant string instead of the path generator function because not all tests use the same screenshot.

So the above configuration stores the baseline, latest, and diff images in the subfolders tests/vrt/baseline, tests/vrt/latest, and tests/vrt/diff respectively. If we provide a common path for baseline, diff and latest images then we should provide some optional configurations such as suffix to distinguish the images. These optional configurations are baseline_suffix, diff_suffix, and latest_suffix.

The threshold option in the visual_regression_settings specifies how sensitive the image comparison will be. To understand what all other options mean, you can check out the README file of nightwatch-vrt github repository

Write Tests

As usual, we start writing our test by making the feature file at first, and the visual check can be done in the step definitions. For the demonstration, I am referencing this project which is a simple to-do app in react.

One of the test scenarios in this app could be as follows:

Scenario: toDo task add test
    Given the user has opened the home page
    When the user adds a new task "New Task"
    Then the task "New Task" should appear in the toDo list
    And the toDo form should match the default baseline

Here, the step And the toDo form should match the default baseline, will perform the visual regression test, and so in the demonstration section, I will be done. And I will focus only on that step.

The implementation of this step can look like the following:

const { Then } = require('cucumber');
const { client } = require('nightwatch-api');

const toDoFormSelector = "#to-do-form"

const assertScreenShot = async function(imgName) {
    await client.waitForElementVisible(toDoFormSelector)
    await client.assert.screenshotIdenticalToBaseline(
        toDoFormSelector,
        imgName,
        `Matched the ${imgName} of toDo page`
    )
}

Then(/^the toDo form should match the default baseline$/, function () {
    return assertScreenShot("todo-form")
});

The assertScreenShot function uses the command screenshotIdenticalToBaseline() that is defined in the nightwatch-vrt library. screenshotIdenticalToBaseline could take up to 4 parameters where only the first parameter is mandatory:

The first one is a String which is the selector of the element that should be tested visually.
The second one is also a String which is the name of the image that is used for the name of the baseline, diff, or latest image. The default name is the name of the selector provided as the first parameter.
The third one is NightwatchVRToptions settings that override the defaults and visual_regression_settings of the nightwatch configuration file.
And the fourth parameter is a String which is the message that is displayed in the console upon the successful completion of the test.

When the VRT is executed for the first time, the baseline images do not exist and so they are created. When the test finds the baseline image with the appropriate name, it will compare the screenshot of the element with the respective image in the baseline. So, if you are running the test for the first time, it will execute twice, once to create the baseline images and again to compare the current state with the baseline. Next time when you run the test, it will be executed only once as the baseline is created only once. However, if there are some changes in the UI and the test fails, then you need to change the baseline image as well.

There are mainly 2 ways to update your baseline image. The easiest way to do this is to set the configuration always_save_diff_screenshot: to true in the nightwatch configuration file's globals section. This will update all the failing screenshots at once. If you need to update only one screenshot then the appropriate method to update it is to delete the baseline screenshot and run the tests again which will take the updated screenshot and save it as the new baseline.

Execute the test

You do not need any special commands to execute the VRT. You can execute the scenario containing the feature file like a normal nightwatch test and that's it. If you need help with setting up nightwatch and running the tests, you can read this blog of mine.

Ride-sharing with geo-fencing technique

Talank — Sun, 21 Mar 2021 13:08:01 +0000

For the undergraduate final year's project, me and my team designed and implemented a taxi ride-sharing algorithm that is targeted at urban areas. In this blog, I am going to describe algorithms that we used for this project and all these algorithms are open source in my GitHub (VroomAlgorithm). If you are willing to further study these algorithms and if you have any confusion, or if you have any suggestions of improvement, then feel free to contact me.

The major algorithms that we used in this ride-sharing system are as follows:

GEO-FENCING

For vehicle searching, we have used the geo-fencing technique. In this technique, we divide the geographical area into multiple smaller areas (based on the district, for example) called ‘Major Fence’. Then we further divide the regions within ‘Major Fence’ into smaller manageable regions called ‘Minor Fence’(for example, city). The size of the ‘Minor Fence’ is determined by the traffic and point of interest. So we convert the vehicle searching problem to a mathematical problem to find if a point lies within a polygon, using the Winding number algorithm. We check if the location of the driver lies within the ‘Minor Fence’ and thus perform the ride-matching. In the figure below, I have shown an example of geo-fences.

Geo-fences Example(showing some fences in different color polygon)

RIDE MATCHING

Ride matching is done with the help of geo-fencing. When a ride request is made in luxury mode(without sharing) or for a motorcycle, then at first, the ride request is forwarded only to the drivers who are available in the same fence. If no free driver is found inside the same fence, then the ride request is forwarded to the drivers in the neighboring fences. So, a single geofence data also contains some information about its 4 neighbors. For example:

name:"fence-name"
topLeftX:"topLeftX-coordinate"
topLeftY:"topLeftY-coordinate"
topRightX:"topRightX-coordinate"
topRightY:"topRightY-coordinate"
bottomLeftX:"bottomLeftX-coordinate"
bottomLeftY:"bottomLeftY-coordinate"
bottomRightX:"bottomRightX-coordinate"
bottomRightY:"bottomRightY-coordinate"
neighbors:{"left-neighbors","right-neighbors","bottom-neighbors","top-neighbors"}

RIDE SHARING

In this project, we used a progressive ride-sharing approach. Whenever a ride offer is created, the system generates the set of geo-fences whose sequence specifies the direction of the ride. The set of geo-fences is generated along the shortest route, between source and the destination, generated by Google Map API. To get the shortest route geo-fences N (the set of geo-fences composing of the route), the only departure and destination location is required. We apply the inverted index data structure on the mechanism of saving the ride’s node. Now whenever the ride request is generated, the system computes the geo-fences along the path of the pickup and drop-off location and checks the following three constraints:

Does the ride pass through the pickup geo-fence?
Does the sequence of geo-fence of the ride offer has the same direction as the sequence of geo-fence of the route between passenger’s pick-up and drop-off location?
Does the vehicle that is offering the ride has enough empty seats?

The figures below show how inverse index data structure can be used for finding path and direction in terms of geo-fences, and how this can be used for shared ride matching.

Inverse Index Data Structure

Shared Ride Matching

Ride Matching and Fence Direction Update

PRE-DEFINED AVAILABILITY CHECK

A normal algorithm might not forward any ride request if the vehicle is full as per it's current situation. However, if we analyse it temporally, then it might be the case that the vehicle will have enough empty seat when it reaches the place(geo-fence) from which new ride request is created. Hece, it is nice to perform a pre-defined availability check.

Let us assume that the number of empty seats available “A” in the vehicle is less than the number of seats requested “R” in the shared ride. Also, suppose that the geo-fence path of the ride matches with the customer’s ride request’s geo-fence path. Now, as per the predefined availability check, we check that how many passengers will be inside the vehicle going through the pickup fence of the requested ride.

Let,

A = Available number of seats,

R = Requested number of seats,

Driver’s updated fence direction = {“N1”, “N2”, “N3”, “N4”...}

Requested ride’s fence direction = {“Nk”,“Nk+1”, “Nk+2”...}

if A <= R at the time of ride request{
 if (Total seat - the number of passengers inside the vehicle 
 who will pass through the geo-fence “Nk”) <= R{
  forward the ride request to the driver
 }
}

Pre-defined Availability Check.

FUTURE DIRECTION

It would be nice to analyze the applicability of this approach. Also in addition to the abovementioned algorithms, temporal analysis (e.g time-series analysis) might be done in the data that is collected every day so that drivers will know which place will have higher ride requests at a certain time of the day. Or even better, the system does so and suggests drivers where to go for more ride requests. If the system does so then the drivers will be evenly distributed in the geo-fences based on the number of potential passengers. However, it might not be as easy as I have said and it is still a topic of research.

REFERENCES

To come up with this algorithm, we took reference from the following papers for learning more about geo-fencing and managing ride sharing using the help of inverse index data structure.

D. Suganthi, S. R. (2018). Vehicle Tracking with Geo Fencing on Android Platform. International Journal of Engineering Science and Computing, 16992-16995.
S. Maximilian, S. H. (2016). A Matching Algorithm for Dynamic Ridesharing. Transportation Research Procedia 19, 272 – 285.

Insight to Visual Regression Testing

Talank — Fri, 12 Mar 2021 04:38:14 +0000

"What you see is what you get" -Flip Wilson

Web UI is advancing every day, and so is test automation. But, there might be several bugs in your application's UI that automated functional testing often misses. Some of these are called visual errors. Visual errors do not mean logical faults in the software. These are the aesthetic flaws that cause interfaces to be displayed incorrectly and make the UX worse. Visual regression tests ensure that there isn't any unintended change in UI. For this, an initial screenshot (called baseline) of the application is captured then, in each successive test, the screenshot of the same component in the current state is compared with the baseline image. If any distinguished changes are encountered, then the test is marked as failed.

It is crucial to know that VRT intends to specify the deviations in appearance only, and failed VRT does not imply that the application is functionally incorrect. For instance, while testing for the responsiveness of a web page, even if automation tools like nightwatch.js can find all the components, it can not be ensured that all those components are aligned properly. So, in the aforementioned scenario, VRT fails even though the function regression test passes.

How does it work?

The primal concept behind VRT is a comparison between two images. Some visual testing tools does pixel to pixel comparison only, whereas some other are more innovative that make use of computer vision to automatically manage baseline. However, UI does not appear exactly the same in all displays and pixel-to-pixel comparison becomes flakier. So to avoid zero-tolerance QA Engineers normally set the threshold to more than 0. In most of the VRT tools such as nightwatch-vrt, the value of threshold ranges from 0 to 1 where a smaller value means more sensitive comparison.

After comparing the latest state of UI elements with the baseline image, if some difference is found by the VRT tool, a difference image is produced so that testers can easily see the difference after the test failure. For example, as shown in the image below, the change in color of the Add button is detected by nightwatch-vrt.

Cons of VRT

By automating the VRT, there will be a rise in maintenance cost as we need to keep screenshots of the web interface updated with respect to different browsers, devices, operating systems, etc.
Image comparison takes more time than normal assertions used in functional testing. Nevertheless, image comparison can lead to false-positive and unreliable results.

Limitation of Automated VRT

CSS animations are very important artifacts for the enhancement of UX. No matter how minor it is, functional animations such as ripple effect in buttons, text animations, animation in loading sign, etc promote user engagements and make an application interactive. But VRT tools that are currently available ignore these animations by either freezing frames, disabling animations, or simply ignoring the CSS transitions.

Tools

There are several free and commercially available tools for VRT. Here, you can find a bigger list of these tools. In this blog, I am going to discuss only 3 tools (nightwatch-vrt, percy, and applitools)

Nightwatch-vrt

Nightwatch-vrt is an open-source library to perform VRT. It is an extension to nightwatch.js with some additional custom commands that compare the screenshot of a DOM element identified by a selector with the baseline image. If there is no baseline image for a particular DOM element, it creates one and uses that for future assertions.

The working mechanism of nightwatch-vrt is quite straightforward. It uses JavaScript Image Manipulation Program Jimp, which is an image processing tool written in Javascript, to compare the current screenshot of a DOM element with the baseline image. The percentage difference in the two images is then compared with the threshold value to get the result of VRT.

To know more about nightwatch-vrt, you can refer to their github repo, or you can stay tuned for my next blog.

Percy

Percy is a commercial tool that can be used for VRT and its working mechanism is pretty advanced than that of nightwatch-vrt. It has a screenshot stabilization technique by freezing animated GIFs on the first frame and disables CSS animation and transition styles. For automated visual testing, web animations are responsible for false-positive results in many cases. There are 3 other methods(Handling dynamic data, Third-party fonts, and ignoring 1px diffs) that Percy uses for stabilization of screenshot and you can read details about it in this documentation. And another advantage of this tool is that it provides SDKs that can be integrated with UI tests in several languages and testing tools like cypress, puppeteer, etc.

Applitools

Like Percy, Applitools is also a commercial platform that provides several functionalities for automated visual testing. Its visual testing library is called Applitools Eyes which can be implemented with different languages like Javascript, Java, PHP, etc. With its visual AI algorithm, Applitools Eyes tries to distinguish visual bugs that are only visible to human eyes. By using the layout algorithm, if testers specify certain regions on the page that contain dynamic contents like date and time, Applitools can check only for layout but not the actual contents. Similarly, if you have some elements in your UI that contain animated GIFs or CSS animations, you can specify those regions as layouts and Applitools will only check for the element to be visible, but ignores the animation.

To learn more about its visual AI algorithms, you can refer to this documentation.

I hope you get some basic ideas about VRT, its pros, cons, and shortcomings. In my next blog, I will show you how to start implementing VRT using nightwatch-vrt. Until then, happy coding.

Testing with Jest and Puppeteer

Talank — Wed, 22 Jul 2020 10:56:45 +0000

"In Jest, there is truth" -William Shakespeare.

By using Jest, you will know the truth about your application. The Jest is a javascript testing tool that is installed via NPM or Yarn and run via the command line. It is a great testing library and many react team members are involved building it, so it happens to work very well for testing react applications. Jest is built on top of Jasmine and Mocha but some additional features like snapshot testing and parallel test execution are provided by Jest. It also comes with built-in mocking and assertion abilities.

Puppeteer is a Node library that is used for browser automation. Puppeteer provides a high-level API to control the browser. It can work with Chrome, Chromium, or Firefox. By default, this library runs the browser in headless mode, but it can be also configured to run Chrome or Chromium fully (non-headless).

Installation And Requirements

This guide is based on Ubuntu 18.04. Some commands will be different if you have another OS installed on your computer. Before getting started with Jest, You need to have npm and an app for which you will be writing tests.

If your app is not yet ready for testing then you can just clone this basic todo app and follow along.

Jest Versions

Jest is the test runner library for creating, executing, and structuring tests. Jest CLI is a tool that you use from the command line to run and provide configuration options to jest. It configures jest based on whatever argument you give while running jest. The version is important because jest is a dynamic library and different versions of jest might work differently. While writing this blog, I am using jest version 24.9.0. So, some features might be different if you are using some other version.

Jest installation and Running Tests

You can install the latest version of jest from NPM using

npm i jest --save-dev

Now, its time to configure the NPM script for running a test from the command line. For this, open your package.json and add the test script as follows:

"scripts": {
    "test": "jest"
  },

Tests are run by using the Jest CLI (typing jest followed by arguments in the command line).

For example, in the todo app in my GitHub, you can run the test with the command npm run test since I have added "test": "jest" in the script of my package.json

With the above-mentioned way, jest can be configured within your package.json. That is the easy way for jest configuration. Alternatively, you can also use the jest configuration file for which you should create a jest.config.js file and include that file in the command to run the test. For example, your package.json scripts section should include

"scripts": {
    "test": "jest --config ./jest.config.js"
  },

And, in the root path alongside the package.json, your jest.config.js should contain the following configurations

module.exports = {
  verbose: true,
  roots: [
    "./__tests__"
  ]
}

By using the jest configuration file, you have many other options as well. You can find the details about that in the jest documentation. However, in this blog, I will focus on configuring jest using the package.json

Install Puppeteer

Use the following command to install puppeteer in your project.

npm i puppeteer --save-dev

Puppeteer will download the latest version of chrome. If you need to work with firefox then you need to install the puppeteer-firefox library. Refer to Puppeteer for Firefox in the npmjs website for further details.

By the end of the installation of jest and puppeteer, you should have the following dependencies in the devDependencies section of your package.json.

"devDependencies": {   
  "jest": "^24.9.0",   
  "puppeteer": "^4.0.1" 
}

Creating Test Files

The jest command runs the test files inside __tests__ folder or it will run any files with .spec.js or .test.js in their filename. So, you have to write your test in a file that ends with .spec.js or test.js. Or you can simply put all your tests inside the __tests__ folder.

Jest Globals

There are varieties of jest Globals but we will mainly need 2 important globals called describe and it. The table below tells about these two globals in more detail.

	it	describe
AKA	Tests	Test Suite
Syntax	it(name, fn)	describe(name, fn)
Description	`it()` global is a method in which you pass a function as an argument, that function is executed as a block of tests by the test runner. You describe your test logic inside `it()` block.	`describe` is an optional method for grouping any number of `it()` or test statements. `describe()` block contains related it blocks.

In addition to it() and describe() methods, you might need the following methods to start testing with jest.

afterAll(fn, timeout)
afterEach(fn, timeout)
beforeAll(fn, timeout)
beforeEach(fn, timeout)

These functions are executed before and after the test scenarios. You can define these functions inside the describe() method to make it applicable for all tests in the scope of that particular describe() method.

Demo:

Here I have added a test for adding a task in my react ToDo app.

const puppeteer = require('puppeteer');

describe('Todo React', () => {
  let browser
  let page

  beforeAll(async () => {
      browser = await puppeteer.launch()
  })

  afterEach(async () => {
      await page.close()
    })

  afterAll(async () => {
    await browser.close()
  })

  describe('add task to the list', () => {
    beforeEach(async () => {
      page = await browser.newPage()
      await page.goto('http://localhost:3000')
    })

    it('should be possible to add task to the list', async () => {
      const taskInputField = await page.$x('//input[@placeholder="Enter task"]')
      const taskToAdd = "New Task"
      await taskInputField[0].click() 
      await taskInputField[0].type(taskToAdd)

      await page.keyboard.press('Enter')

      const lists = await page.$x("//div[@class='list']/p/input");

      let toDo;

      for( let list of lists ) {
        toDo = await page.evaluate(el => el.getAttribute("value"), list);
      }

      expect(toDo).toBe(taskToAdd)
    })
  })
})

Explanation of Demo

In the above demo, I first imported the puppeteer library. And in before scenario, I launched the browser in headless mode. Before all the scenarios inside the first describe block, the browser is launched only once. And before each scenario, a new tab is opened in incognito mode. Similarly, after each scenario, the tab is closed and when all the scenarios are executed, the browser is closed.

The browser is launched in headless mode by default. However, if you want to launch the browser in non-headless mode then you can provide the browser launch option in puppeteer.launch(). For example:

browser = await puppeteer.launch({headless:false})

In Jest, there are plenty of functions to simulate UI activities. For example, in the above demo, you can see click() for clicking in an element, type() for typing, etc.

Similarly, for assert operation, you can use expect(). Many other matcher functions can be used with expect() such as .toContain(item) , toBeTruthy() etc. You can find more of these jest matchers in the jest documentation.

Gherkin with Jest

In Jest, you do not have to write a feature file for your test cases. If you need to write features in a separate file then you can use jest-cucumber library. By using this library, you can replace describe and it blocks by Given, When, and Then step definitions. The step definitions are linked with the respective steps in the feature files.

However, I would prefer Codecept.js or nightwatch.js instead of jest if I need to use features of Cucumber.js while testing my application. Because, with jest-cucumber, you need to specify which feature file is linked with a particular step definition file. While Codecept.js or nightwatch.js, you can write the step definition of a given step in one file and the when step of the same scenario in some other file. You might lose this kind of freedom if you are using jest-cucumber.

That's it for now. If you want to learn more about jest and it's commands, you can refer to jestjs.io

Hope you enjoy jesting.
ThankYou!

Setup Nightwatch and run your first test

Talank — Tue, 18 Feb 2020 05:50:28 +0000

Nightwatch.js is an open-source automated testing framework that aims at providing complete E2E (end to end) solutions to automate testing with Selenium Javascript for web-based applications, browser applications, and websites. It is written in Node.js and uses the W3C WebDriver API (formerly Selenium WebDriver) for interacting with various browsers.

Installation And Requirements

This guide is based on Ubuntu 18.04. Some commands will be different if you have another OS installed on your computer. Also, we will be following the BDD approach in this blog. So, if you want to learn more about BDD first then please read our blog on Demonstrating BDD (Behavior-driven development) in Go. Before getting started with Nightwatch we need to have the following installed on our computer.

1. Node.js:

To install Node.js:

sudo apt install nodejs

To verify the successful installation of nodejs and to check its version use the following command

nodejs -v

2. Node Package Manager(npm):

The nodejs package contains both the node and npm binaries. So, npm is also installed when you run the command to install nodejs. However, to verify that npm was also successfully installed, you can check the version of npm by running the command

npm -v

3. Java:

We need java to run the Selenium server which automates our browser. If your machine does not have java installed then install it using the following command

sudo apt install default-jdk

If you want to learn more about how to install Java, then you can follow this link.

4.Selenium Server:

Download the latest stable version of the Selenium standalone server JAR file.

5.Chrome Driver:

Lastly, download the latest stable version of Chrome Driver.

Once you have downloaded Chrome Driver, you need to unzip it by running the following command:

unzip chromedriver_linux64.zip

Once you have unzipped it, you need to move the chromedriver(shared library) and place it inside the same folder where you have placed the Selenium standalone server file.

Setting up Nightwatch.js

Make a configuration file and name it nightwatch.conf.js inside the root directory of your project and inside the configuration file add the following content.

module.exports = {
    src_folders: ['test'],
    test_settings: {
        default: {
            selenium_host: '127.0.0.1',
            launch_url: 'http://localhost:3000',
            globals: {},
            desiredCapabilities: {
                browserName: 'chrome',
                javascriptEnabled: true,
                chromeOptions: {
                    args: ['disable-gpu'],
                    w3c: false
                }
            }
        }
    }
}

You can also make a nightwatch.json configuration file as an alternative to the nightwatch.conf.js file. Please refer to the nightwatch website to learn how to write the nightwatch.json configuration file.

You need to define src_folders and launch_url inside the nightwatch.conf.js where src_folders is used to define the folder in which you have written your tests and launch_url is used to define the index URL of your project which is used as the main URL to load and run your tests.

src_folders and launch_url will be different for different projects.

src_folders: YOUR_SOURCE_FOLDER_FOR_TEST_FILES
launch_url: URL_OF_YOUR_WEBSITE

Run this command to install nightwatch and cucumber into your project dependencies which allow us to run automated tests using the gherkin language.

yarn add --dev nightwatch-api nightwatch cucumber

Or,

npm install --dev nightwatch-api nightwatch cucumber

Now the following information should be added inside package.json in devDependencies. But do remember that these versions may also vary depending upon the releases of these dependencies.

 "devDependencies": {
    "cucumber": "^6.0.5",
    "nightwatch": "^1.3.3",
    "nightwatch-api": "^3.0.1"
 }

You can use npm or yarn, whichever you prefer. However, if you want to use yarn but do not have it already installed then, you can install it by downloading the .deb file from the yarn releases and run the following command:

 sudo dpkg -i yarn[VERSION].deb

The .deb file might not be compatible with your machine. It only works for debian based linux distros.

For the rest of the setup process, I am using npm.

Now, create index.js inside [root_dir]/test/acceptance and add

const { setDefaultTimeout, After, Before } = require('cucumber')
const { createSession, closeSession, startWebDriver, stopWebDriver } = require('nightwatch-api')

setDefaultTimeout(60000)

Before(async () => {
    await startWebDriver();
    await createSession();
})

After(async () => {
    await closeSession();
    await stopWebDriver();
})

Here, we are using before() and after() hooks which will allow us to execute the code inside them before and after each test scenario.

Before starting the Selenium server you need to change your directory to where the selenium and chromedriver files are. After doing that you can now start the Selenium server with the command:

java -jar selenium-server-standalone-3.141.59.jar -port 4444

Testing your Project

Create feature files inside [root_dir]/test/acceptance/features with an extension .feature for each feature of your project. Follow the Gherkin reference to write your feature file.

We do not need to use gherkin language but I am using it because this is the same language that user and domain experts use while they talk about the domain. Also, it is easy to understand for both technical and non-technical personnel involved in the project.

A feature file written using gherkin language looks like this.

Feature: search using transliterated Latin characters
  As a user
  I want to find the literature by specifying Latin (English-style) script equivalent to the Nepali
  So I can find the literature without needing to use a Nepali keyboard

  Background:
    Given the user has browsed to the homepage

  Scenario: user should be able to search in English characters using transliterated text
    When the user enters the English characters "munaa madan"
    And the user selects the transliterated text
    And the user searches for the literature
    Then the search result should be displayed

Next, create a javascript file for the step-definitions inside [root_dir]/test/acceptance/stepDefinitions for each feature. For each step in the scenario of your feature file, a matching step definition is executed while running the test. Step definitions for the above search feature will look something like this.

const { Given, When, Then } = require('cucumber');
const { client } = require('nightwatch-api');
const searchField = "//input[contains(@class, 'SearchBox')]";
const searchButton = "//button[contains(@class,'SearchBox__Button')]";
const searchResultFoundContainer = '.sk-hits';
const searchResultNotFoundContainer = '.sk-no-hits';

const userEntersTheEnglishOrNepaliCharacters = (searchTerm) => client.useXpath()
    .waitForElementVisible(searchField)
    .setValue(searchField, searchTerm)
    .useCss();

const userSearchesForTheLiterature = () => client.useXpath()
    .click(searchButton)
    .useCss();

Given('the user has entered the English/Nepali characters {string}', userEntersTheEnglishOrNepaliCharacters);
When('the user enters the English/Nepali characters {string}', userEntersTheEnglishOrNepaliCharacters);

Given('the user has searched for the literature', userSearchesForTheLiterature);
When('the user searches for the literature', userSearchesForTheLiterature);

Then('the search result should be displayed', () => client.waitForElementVisible(searchResultFoundContainer)
    .waitForElementNotPresent(searchResultNotFoundContainer));

Then('the search result should not be displayed', () => client.waitForElementVisible(searchResultNotFoundContainer)
    .waitForElementNotPresent(searchResultFoundContainer));

Finally, run the test from the root directory using the command:

cucumber-js --require test/acceptance/index.js --require test/acceptance/stepDefinitions test/acceptance/features/[YOUR_FEATURE_FILE].feature

Does the above command look long? I am sorry, but you have to use that command every time you run the test. But only the name of your feature file is different, while running different tests, the rest is all the same. So, let's assign an easy short name for the rest and put it inside the scripts section of package.json. I called it test-e2e and inside the scripts of package.json, I added

"test-e2e" : "cucumber-js --require test/acceptance/index.js --require test/acceptance/stepDefinitions"

Now, the above long command is shortened to

npm run test-e2e test/acceptance/features/[YOUR_FEATURE_FILE].feature

That's it. Now, you are ready to run test suites for your project. You can refer to Nightwatch to read more about Nightwatch and its commands.

Hope you enjoy testing.
ThankYou!