Forem: Beanworks

The Painful Parts of End-to-End Test Automation for your Windows Application

pauliedoherty — Wed, 27 Jan 2021 21:58:39 +0000

Overview

This post is targeted at all you test engineers who do not want to add C# and all the wonderful Visual Studio tools that come with it to your test automation stack. You’re thinking, since our web app and mobile app’s e2e testing are already implemented with NodeJS why should testing our windows app be any different?

Here are the weapons we will be wielding to achieve this:

A minimum of Windows 10 or Windows Server 2016 desktop
An automation tool to drive your application under test (AUT) - WinAppDriver & Appium
A compatible automation client - WebdriverIO
A test runner - Mocha
A tool to run tests without manual user logon - paexec
An Automation Server of sorts (details of this are out of scope of this post)

Other handy extras:

...Note: Many parts in this post are only vaguely covered because there are already so many great resources out there. The parts that are described in detail were difficult to find good resources on, hence this post...

Pain Point 1 - Preparing Test Environment for Automation (lil' painful)

WinAppDriver & Appium:

WinAppDriver is Microsoft's in-house developed tool to automate Universal Windows Platform (UWP), Windows Forms (WinForms), Windows Presentation Foundation (WPF), and Classic Windows (Win32) applications and is developed with Appium integration in mind. WinAppDriver is what does the heavy lifting behind the scenes. It carries out our mouse clicks and keyboard presses as we request. The image below shows the communication flow of our solution:

Appium is a RESTful server that acts as a wrapper between WinAppDriver and your automation client. Appium accepts commands from WebdriverIO and forwards them to WinAppDriver. In response to a request, it returns a status code and logs to the automation client.

To run WinAppDriver & Appium on our test environment we need to:

Download and install WinAppDriver
Enable Developer mode on Windows
Install NodeJS

With these prerequisites installed you are ready to automate using WebdriverIO’s test runner - wdio.

Pain Point 2 - Configuring WebdriverIO and running your first test (somewhat painful)

WebdriverIO

WebdriverIO (wdio) is a Selenium-like client service that sends requests to Appium via W3C WebDriver Protocol. This is the framework we will use to write our test cases and specify how they will run.

Before proceeding, please initialize a nodejs project in your users home directory. If you're unsure how to do this Phil Nash's guide is excellent. I have creatively decided to call my project my-project.

Taking from WebdriverIO's guide let's set up the test-runner by opening the Command Prompt and running:

mkdir my-project & cd my-project
npm install @wdio/cli
npx wdio config

This will launch a helper utility where you choose what services and project structure you wish to use when running wdio. To get things started please mirror these selections:

Your project file tree should look like:

wdio.conf.js is the file that holds the configuration properties of the entire communication flow diagram in the first section. It is now loaded with the properties needed to allow communications between our webdriver client and Appium server. However, we still need to define our capabilities. Capabilities define which environment we are running Appium on, which protocol Appium will use, and what AUT it will be automating. Let’s uphold tradition and use the classic calculator example. Below are the capabilities required to connect Appium to the calculator application on Windows Server 2019 along with other properties relevant to this post:

// wdio.conf.js snippet
    runner: 'local',
    path: '/wd/hub',
    port: 4723,
    baseUrl: 'http://localhost',

    capabilities: [{
        platformName: 'windows',
        automationName: 'windows',
        deviceName: 'WindowsPC',
        app: 'C:\\Windows\\System32\\win32calc.exe'
    }],

    services: [
        ['appium', {
            logPath: './'     // logs are friends
        }]
    ],

    framework: 'mocha',

    specs: [
        './src/test/specs/**/*.js'
    ],

A few things to note here -

Defining appium in services instructs wdio to start an Appium server
Runner, path, port and baseUrl properties are used to connect wdio client to the host Appium server
Defining capabilities tells Appium what application to automate and what automation tool we want it to configure; WindowsPC maps Appium to WinAppDriver
We define mocha as the test framework and
We tell said framework what location the spec file are in

More details on Appium/WinAppDriver can be found here.

Side Note:

I highly recommend using Appium-Desktop as a development tool to troubleshoot any Appium connectivity issues. You can download it here.

Start Appium Desktop from Start Menu, open File -> New Session Window… and enter the capabilities above in the 'Desired Capabilities' tab:

Click Start Session to connect Appium to calculator app:

If you cannot connect to your AUT with capabilities in Appium-Desktop, you will also not be able to connect through wdio, so make sure to get things working here first. Here's a contrived error on Appium-Desktop to give an example of error logs:

Now with our capabilities sorted we can start writing test cases for the wdio client.

Pain Point 3 - Controlling app elements (pretty painful)

Shifting attention to our test spec file ./test/spec/example.e2e.js, we can define code to send requests to the Appium server. One of the beauties of webdriver is that the client object is accessible anywhere in wdio’s scope so we can directly call its properties. Let’s do some quick math to verify our calculators sum() functionality => 2 + 2 = 4

// ./test/spec/example.e2e.js
describe('Quick math', () => {
    it('Two plus two is four', () => {
        $('~132').click();              // click 2
        $('~93').click();               // click +
        $('~132').click();              // click 2
        $('~121').click();              // click equals
        const result = $('~150')        // get result
        expect(result).toHaveText('4 ') // verify it equals 4 
    });
});

A few things to note here:

Test spec files are using mochajs
Webdriver uses expect for assertions
The selector method uses ~ to identify the AutomationId. There’s not much clear cut documentation on this but you can work back from here to figure it
Yes, I did append a space on the 4

We are finally ready to run our tests! In the Command Prompt run

npx wdio

Here’s the result from npx wdio without appending a space:

And here it is with appending a space:

OK, I cheated. I should have parsed the data before asserting. But I refuse to apologize.

Side Note:

Use inspect.exe to find the AutomationId of your elements. Open inspect.exe and hover mouse over element you wish to inspect. Here’s an example of the ‘2’ button on the calculator app:

Thus concludes setting up automation of a windows desktop application.

On a side note, here's a great example of a scalable, maintainable wdio project using the Page Object Model pattern.

Pain point 4 - Running WDIO in a CI/CD pipeline without manual access to the GUI (painful)

What we have achieved so far is great for development and presentation purposes. But to provide real value and execute this in a CI/CD pipeline we will need some extra setup in our test environment. Keeping cost saving in mind, we will not want our test environment running when tests are not being executed. It also becomes clear that Remote Desktop (RDP) is not very useful when it comes to automated pipelines. This now presents an interesting challenge - how do we run our tests with no user interaction or manual login step? One way is to send the test command over ssh. This is what we will explore.

Aside: configuring communication to the test environment is out of scope of this post; there are lots of great articles on setting up ssh servers out there.

This part assumes we are running our test environment in a cloud based CI/CD pipeline, therefore the test environment is not being accessed through the GUI via RDP nor is it sitting beside you on your desk. With all that in mind, let’s remotely run our test cases.

ssh ${USER}@${IP_ADDRESS} 'cd my-project & npx wdio'

Hmmmm, we are met with a pleasant 500 server error.

Let’s see what’s going on by checking our processes in parallel while wdio is attempting to run:

ssh ${USER}@${IP_ADDRESS} 'tasklist /fi "imagename eq node.exe" & tasklist /fi "imagename eq win32calc.exe"'

This gives us:

Hmmmm, node.exe and win32calc.exe are running as services on Session# 0. Service processes on Session# 0 are reserved for user agnostic processes and do not run with a GUI. Therefore Appium/WinAppDriver has no GUI output for it to find the running calculator application which also doesn't have a GUI output. This is sad times for people hoping to test the GUI.

Let’s take a step back to figure out what's going on. Let's open an RDP connection to our test environment. Here, we can directly run our tests, and in parallel check our running processes.
Using the Command Prompt (cmd) on the test environment let's run

cd my-project & npx wdio

And at the same time in a separate terminal run:

tasklist /fi "imagename eq node.exe" & tasklist /fi "imagename eq win32calc.exe

We see the tests have ran successfully:

And our processes are showing an RDP Session which does use the GUI. All actual logged in users get assigned Session# > 0.

To solve the issue in our CI/CD pipeline we must configure our test environment to run our code as a logged in user. This will allow node.exe and win32calc.exe to run as a GUI Session.

To achieve this, we must do 2 things:

Enable Autologon &
Use paexec to run processes as the System account user

This solution is stated by hepivax on this thread.

Enable AutoLogon:

Please use guide here.

PAExec:

PAExec is an open-source equivalent of PsExec

PAExec allows you to run interactive command-prompts on local and remote servers as the System user account (it can do a lot more too!).

Let’s install it on our test server and add it to a System Variable location so we can run it from anywhere with cmd:

Download paexec.exe
Copy and paste to C:\Windows\paexec.exe (If you’re unsure of this step refer to this)
Test cmd has access to paexec.exe by running

where paexec

it should return C:\Windows\paexec.exe

You can run paexec in command prompt for more details on its use cases.

Now that PAExec is installed and accessible, we can open a Command Prompt with PAExec so that cmd is running under System user. Let's do this and verify where we are in the file system by running:

paexec -s cmd /C "cd"

Breaking this command down:
paexec -> Executes paexec.exe
-s -> Flags PAExec to run requested process under System user
cmd -> Open Command Prompt
/C -> Tells PAExec to pass a command to Command Prompt
“cd” -> Once paexec runs Command Prompt as System user it executes cd which returns the current directory path

This outputs our System account user path, verifying we are running cmd as System user:

Now any command we choose to execute in the Command Prompt with PAExec will run under the System user

...piecing it all together
Over in our test automation server (or whatever you are using to remotely trigger your test suite), run the following:

ssh ${USER}@${IP_ADDRESS} 'paexec -s cmd /C "cd C:\Users\${USER}\my-project & npx wdio"'

Note: Remember the new path is required here because we opened a cmd window from the System user account with PAExec.

...And in parallel check that node.exe and win32calc.exe are running as a logged on user with GUI access:

ssh ${USER}@${IP_ADDRESS} 'tasklist /fi "imagename eq node.exe" & tasklist /fi "imagename eq win32calc.exe"'

Let's look at our processes:

We see they are running under a Session# > 0 and Voila! Test executed successfully.

PHP For The Impatient

Tom Lei — Fri, 25 Sep 2020 04:04:43 +0000

We are primarily a PHP shop for about half of the engineering team here at Beanworks. PHP tends to have a bad reputation out there, but with PHP 7 (and potentially PHP 8), it's really becoming a mature and powerful language while keeping its simplicity.

I always tell new engineers with no prior PHP experience that if you're able to write code in any language, you will be able to learn PHP pretty quickly.

In this article we will explore some PHP syntax and compare it to Javascript or Typescript.

I have added 🔁 icon to the ones that have identical syntax.

Basic Syntax

Variables

In PHP land, all variables have to start with a dollar sign $

PHP

$foo = 'bar';

NodeJS

let foo = 'bar';

Template Literals

You can use variables in strings as long as the string is wrapped with double quote ("), strings wrapped with single quotes will not evaluate the variables.

PHP

$str = "My name is {$name}"

NodeJs

let str = `My name is ${name}`

Constants 🔁

PHP has real constants, they must be capitalized, but otherwise the syntax is actually the same

PHP

const FOO = 'bar';

NodeJS

const FOO = 'bar';

Arrays

PHP

$arr = ['a', 'b'];

// you can iterate using foreach
foreach ($arr as $element) {
  ...
}

NodeJS

let arr = ['a', 'b'];

// you can iterate using for ... of
for (let element of arr) {
  ...
}

Objects

In PHP land, objects are just arrays, associative arrays. Instead of using :, you need to use a double arrow =>

PHP

$myFavouriteVehicles = [
  'Toyota' => ['4Runner','Tacoma'],
  'Ford' => ['Bronco']
];

$fordVehicles = $myFavouriteVehicles['Ford'];

// you can iterate key => value using foreach
foreach ($myFavouriteVehicles as $brand => $vehicles) {
   ...
}

Javascript

let myFavouriteVehicles = {
  'Toyota': ['4Runner','Tacoma'],
  'Ford': ['Bronco']
}

let fordVehicles = myFavouriteVehicles['Bronco'];

// you can iterate key value with Object.entries
for (let [brand, vehicles] of Object.entries(myFavouriteVehicles)) {
  ...
}

Callback functions

In most of the PHP land, you still have to write call back functions like you had to in ES5 days, arrow function is only recently supported in PHP7.4

PHP 🔁

array_map(function($element) {...}, $arr);

// or in PHP 7.4+ you can use arrow function, no multi line though
array_map(fn($element) => ..., $arr)

NodeJS

arr.map(function (element) {...});

// Arrow function since ES6
arr.map(element => ...);

Classes & Inheritance

PHP's uses the same syntax extending classes and interfaces

PHP 🔁

class Foo extends Bar implements FooBarInterface

Typescript

class Foo extends Bar implements FooBarInterface

Type system

Contrary to popular beliefs, PHP does have a type system that works similar to other OOP languages, and with each new version of PHP, the type system is getting better.

PHP

public function writeLine (string content, LineWriter $writer) : string {
    return content;
}

Typescript

public writeLine( content : string, lineWriter lineWriter) : string {
    return content;
}

Invoke Methods

To invoke a method, you need to use -> instead of a dot in PHP.

PHP

$logger->log('PHP uses ->');

NodeJs

logger.log('Node uses .');

Generators 🔁

There is no special syntax in the function signature if you want to use the yield keyword

PHP

public function getLaptopBrands() : Iterator {
  yield 'Lenovo';
  yield 'Asus';
  yield 'Dell';
}

TypeScript

public function* getLaptopBrands() : IterableIterator<string> {
  yield 'Lenovo';
  yield 'Asus';
  yield 'Dell';
}

Yield from generator

In PHP you can use the yield from syntax to yield from a generator

PHP

public function getComputerBrands() : Iterator {
  yield from getLaptopBrands();
  yield from getDesktopBrands();
}

TypeScript

public function* getLaptopBrands() : IterableIterator<string> {
  yield* getLaptopBrands();
  yield* getDesktopBrands();
}

GraphQL Schema Test with GraphQL Inspector

Katelyn Truong — Thu, 30 Jul 2020 23:52:23 +0000

Background

We were nearing the alpha release of one of our new modules which includes the release of a new mobile application. Our web app, new mobile app and our desktop app, all consume our graphQL API.
To increase the reliability of our platform, we needed to formalize and implement a backwards compatibility and deprecation policy for our GraphQL API.
The rules to add or delete from schema were

Only add new fields.
If you want to remove a field, deprecate it and remove it when it’s no longer in use.
If you want to change an existing field, create a new field, deprecate the old field and remove it once it’s no longer in use.

However, since the mobile app has a much larger footprint as users might be using the app with the older version with old graphQL schema, having a test for it seemed inevitable. Our existing e2e automation on mobile did not cover all graphQL endpoints, and it would have been too expensive and time consuming to cover all endpoints from a user interface perspective. Moreover, we wanted backward compatibility test to be a test that runs before we run e2e tests to detect breaking changes earlier in the cycle.

Since we needed something light weight and graphQL compatible, we decided to use GraphQL Inspector Tool to ensure backward compatibility and prevent schema breaking changes.

Tech Stack

Bitbucket (Git)
Codeship Pro for CI/CD workflow
Docker
GraphQL

Code Structure

Why GraphQL Inspector Tool?

We decide to use GraphQL Inspector Tool for graphQL backward capability test because of following reasons:

GraphQL compatibility
Ease of implementation with custom rules
Easy CLI command for CI integration

Steps to Implement

Step1: Create a dockerfile for GraphQL Inspector

Dockerfile template can be found here at graphQL inspector docker hub
Dockerfile can be named as Dockerfile.graphqlInspector in ./codeship folder

FROM node:10-slim

ENV LOG_LEVEL "debug"
ARG PRIVATE_SSH_KEY

RUN apt update 
RUN apt install -y ssh
RUN yarn global add @graphql-inspector/cli@2.1.0 graphql
RUN apt install -y git-all

# Copy the decrypt ssh private key for Bitbucket Readonly User to allow codeship access to 
# Our repository from codeship container  
RUN mkdir -p /root/.ssh
RUN echo $PRIVATE_SSH_KEY >> /root/.ssh/id_rsa
RUN chmod 600 /root/.ssh/id_rsa
RUN ssh-keyscan -H bitbucket.org >> /root/.ssh/known_hosts

RUN mkdir /app
WORKDIR /app

Step2: Create an entrypoint file

This file will be triggered after docker image is built
Name file entrypoint-graphql-inspector.sh in ./codeship folder
- To add git remote for our repository
  - Note: it uses remote name as api because origin already taken and it doesn't work in codeship
- To fetch the master branch which will use for graphql inspector command in codeship

#!/bin/bash
git remote add api git@bitbucket.org:<owner_name>/<repo_name>.git
git fetch api master
exec "$@"

Step3: Create graphQL inspector test service

graphql-inspector-test service will be placed in ./codeship-services.yml
Build from Dockerfile.graphqlInspector
Include entrypoint link with entrypoint-graphql-inspector.sh to trigger the command line after finishing pulling and building the docker image from dockerfile
For volumes, it can be different depending on your code structure or same as created above in dockerfile.
- How to find which volume to mount
- Install jet if you have not (for locally running commands for codeship)
- cd <repo>
- jet run SERVICE_NAME pwd
- Example: jet run graphql-inspector-test pwd
- With pwd, it will show the path of the volume where it should be mounted

services:
  graphql-inspector-test:
    build:
      context: ./
      dockerfile: ./codeship/Dockerfile.graphqlInspector
      encrypted_args_file: <path_to_encrypted_git_ssh_key>
    encrypted_env_file: <path_to_encrypted_git_ssh_key>
    entrypoint: ["sh", "./codeship/entrypoint-graphql-inspector.sh"]
    cached: true
    volumes:
      - .:/app

Step4: Create graphql-inspector-test step

GraphQL Schema Inspector Test test step under Jest Tests in ./codeship-steps.yml
Use graphql-inspector-test service
To trigger command graphql-inspector diff git:api/master:./src/generated/graphql/schema.graphql ./src/generated/graphql/schema.graphql --rule ./src/generated/graphql/custom-rule.js

- type: serial
  name: Continuous Integration
  encrypted_dockercfg_path: dockercfg.encrypted
  steps:
  - name: Tests
    type: parallel
    exclude: ^(production)
    steps:
    - name: Jest Tests
      type: serial
      steps:
      - name: Jest Tests for E2E Tests
        service: e2e-runner
        command: yarn test
      - name: ESLint for E2E Tests
        service: e2e-runner
        command: yarn run lint
      - name: Graph QL Test
        service: graphql-test
        command: yarn run graphql-test
      - name: Graph QL Schema Inspector Test
        service: graphql-inspector-test
        command: graphql-inspector diff
                 git:api/master:./src/generated/graphql/schema.graphql             
                 ./src/generated/graphql/schema.graphql --rule 
                 ./src/generated/graphql/custom-rule.js
  - name: Deploy to Dev and Production

Step5: How to create custom rules for graphQL inspector test

For details, you can see graphql inspector custom rules
We use custom-rules to skip certain attributes that we don’t want the test to run
- Example: the field may not be available to mobile apps yet and still under development, so there may be many changes to the field.
Create file name custom-rules.js in the folder where you have your schema.graphql

// custom-rules.js
/**
 * @summary exceptionSet is an array of schema attributes that can be skipped during graphql schema diff test
 * for breaking changes. 
 * !important: by adding to exceptionSet, there is a risk of graphql schema test will skip these attributes
 * example: new Set(['Form.user]);
 */
const exceptionSet = new Set([]);

/**
 * @summary changes is an object array of all breaking changes [{error1},{error2}]
 * Each error object contains these key value pairs as follow 
 * critical: { level: 'BREAKING', reason: "some reason"}
 * type: 'FIELD_TYPE_CHANGED' or 'FIELD_REMOVED' or 'ENUM_VALUE_REMOVED', etc... 
 * message: "Field 'XYZ' was removed from interface 'ABC'"
 * path: graphql schema attribute, eg: 'Comment.description' 
 */
module.exports = ({ changes }) => {
    const included = [];
    const excluded = [];
    for (const change of changes) {
        if (exceptionSet.has(change.path)) {
            excluded.push(change);
            continue;
        }
        included.push(change);
    }
    if (excluded.length > 0) {
        const warnColor = '\x1b[33m';
        console.log('The following changes have been excluded from breaking changes test:');
        for (const change of excluded) {
            // print something sensible - should show that the field was excluded
            // and what potential problem we're ignoring ;-)
            const message = `!!! ${change.path}: ${change.message}`;
            console.log(warnColor, message);
        }
    }
    return included;
};

How to execute test locally

Prerequisite

npm install --global @graphql-inspector/cli graphql

Steps

cd <repo_folder> 
graphql-inspector diff git:origin/master:./src/generated/graphql/schema.graphql ./src/generated/graphql/schema.graphql --rule ./src/generated/graphql/custom-rule.js

Conclusion

We were successfully able to add graphQL backward capabilities test that is easy to maintain, and is platform agnostic by using GraphQL Inspector Tool. We are able to gain a better understanding of how API should be handled for projects, where the same API is used across multiple platforms (i.e. web, desktop and mobile).