Forem: Jochen

Optimising Appium Tests

Jochen — Sat, 28 May 2022 07:31:49 +0000

Let's first begin by recapping what Appium is, what it does and why it is used by developers and QA for both mobile web testing and mobile app testing.

What is Appium?

Appium is an open-source framework, built in Javascript, which provides a collection of tools allowing you to run automated tests on mobile simulators/emulators and physical mobile devices.

Automated tests are test scripts which perform various actions on a website, or mobile app, while checking if the result of those actions are the intended result.

Both website and mobile app developers, as well as QA departments, use automated testing as a way to find regression bugs or incompatibilities on various devices.

How do I use Appium?

You can easily set up Appium if you have already installed NodeJS. To install, simply run this command:

npm i appium

Next, make sure you have either an iOS simulator or Android emulator installed on the same system. You can also connect a physical device to the machine, through USB, which should work just as well with Appium.

Once setup, you can write your first Appium test. In this example, let's use WebDriverIO:

npm install webdriverio

Now, copy/paste the example code below to run your first test:

const opts = {
  path: '/wd/hub',
  port: 4723,
  capabilities: {
    platformName: "Android",
    platformVersion: "12",
    deviceName: "Android Emulator",
    app: "https://github.com/appium/appium/raw/master/sample-code/apps/ApiDemos-debug.apk",
    appPackage: "io.appium.android.apis",
    appActivity: ".view.TextFields",
    automationName: "UiAutomator2"
  }
};

async function main () {
  const client = await wdio.remote(opts);
  const field = await client.$("android.widget.EditText");
  await field.setValue("Hello World!");
  const value = await field.getText();
  assert.strictEqual(value, "Hello World!");
  await client.deleteSession();
}

main();

This will start an automated test session on an Android 12 emulator, running on your machine. It will open a demo app, input 'Hello World' in an input field and check if the input was successful.

Optimising Appium Tests

Appium provides various options, out of the box, that make testing as stable as possible on various devices.

There are some settings you can tweak, to make Appium more performant.

These settings depend on wether you want to run tests on Android or iOS. Let's see a list of settings we can modify to make Appium faster:

Cross Platform Capabilities

noReset: By default Appium will reset as much as possible in between sessions. If you do not want this, setting this to false will speed up test execution.
fullReset: Only use this if you really have to. Usually not required
isHeadless: Setting this to true will start a simulator or emulator without an UI. This should be faster.

Android Capabilities

autoGrantPermission: Setting this capability to true allows Appium to determine your app permissions and grant them at the start of the test.
appWaitPackage / appWaitActivity: Indicate to Appium which activity is supposed to start. This prevents having to wait for Appium to inspect the apk and loop over activities.
skipUnlock: Appium uses a helper tool to check if the device or emulator under test is locked, for example with a pin code. If this is not the case, skipping this will shave off a few seconds.
skipDeviceInitialization: Setting this to true will cause Appium not to install the io.appium.settings helper app.
skipServerInstallation: If the server app is already on the device, skipping this will shave off a second or so.

iOS Capabilities

iosInstallPause: Let Appium know how long to wait before starting the test.
maxTypingFrequency: This capability specifies the maximum speed when typing on an iOS device or simulator.
realDeviceScreenshotter: This will use the idevicescreenshot program, which is part of libimobiledevice, to capture screenshots. It should be more performant.
simpleIsVisibleCheck: This will use a simpler visible check than the default one used by WebDriverAgent.

Appium Cloud Testing

Once you want to run Appium automated tests on multiple devices, you might want to consider using a cloud provider. The advantage of this is that these provide multiple physical devices, iOS and Android, which are already preconfigured and ready to run your tests.

You will get instant test results, including a video, screenshots and logs. TestingBot is an Appium cloud provider which offers access to physical Android and iOS devices for native mobile app testing.

Visual Regression Testing with Playwright

Jochen — Mon, 20 Sep 2021 12:12:25 +0000

What is Playwright?

Playwright is a NodeJS library, created by Microsoft, which offers very similar features to Puppeteer. Both libraries allow you to automate actions within a browser.

With Playwright, you can start or connect to a Chrome, Edge, Safari or Firefox browser and send commands back and forth. The protocol used for these messages is the DevTools protocol (for Chromium browsers) and a custom protocol for Firefox and WebKit browsers.

The main advantage of using Playwright instead of Puppeteer is that Playwright supports multiple browser vendors, where as Puppeteer only supports Chromium based browsers.

How can I use Playwright to do automated testing?

One of the use-cases in using a browser automation framework, is to use it with a testing framework. Popular testing frameworks, such as Jest and Mocha, already implement support for Playwright testing.

These test frameworks allow you to start or connect to a browser and perform the actions defined within your test cases.

An important thing to keep in mind is that Playwright will run in a headless-mode by default. This means that the UI of the browser will not be visible to the user. While running your automated tests, you will not see the browser perform these actions. Headless mode is used to speed up the Playwright execution: it reduces CPU usage because no UI updates are required and improves the speed of total execution.

What is Visual Regression Testing?

Visual Regression Testing is used to prevent unintentional changes to an application's visual state. Let's say you have a website and you know how the page should look like. You instruct your test framework to take a screenshot of your pixel-perfect page, and tell the framework that the page should always look identical to the screenshot you just took.

In a perfect world, your webpage will always look perfect to the end user, just like the screenshot you took. However, sometimes an update to your website, by a developer or someone else in your team, might cause a (visual) bug. This could be a DOM element missing on your page, or a CSS change that causes rendering issues, or any other bug resulting in an inferior experience to the website visitor.

This is where the power of Visual Regression Testing shines: it will alert you when an unintended change occurs on your website and causes a visual bug.

Visual Testing Libraries

There are quite some libraries available online, open-sourced and other licensed visual comparison libraries, that help you to set up and perform regression testing.

Popular libraries such as BackstopJS for Puppeteer, or Loki provides developers and QA with powerful techniques to quickly, and easily, set up visual regression testing.

In the case of Playwright testing, Playwright Test provides a built-in Visual Comparison feature to take, and visually compare, screenshots with Playwright.

Configure Visual Comparison Testing with Playwright

Playwright contains a package called Playwright Test, which is built as an utility to run automated tests with Playwright.

When you first run a test, Playwright Test will take a screenshot of the page, a baseline screenshot (or golden image), and save it in a folder.

Subsequent tests will continue to take screenshots and match these, pixel-by-pixel, with the baseline screenshot.
Let's say you saved your golden image as golden.png, a simple assertion would be to use this syntax:

expect(await page.screenshot()).toMatchSnapshot('golden.png');

Where page is an object provided by Playwright Test, trying to match the output of page.screenshot() with the contents of golden.png.

When a test fails, meaning the page has changed its contents, you will receive a test error, indicating the screenshot no longer matches the golden image.

Internally, Playwright Test uses the pixelmatch library to compare screenshots for visual differences. It is possible to pass several options to this library, to fine-tune the comparison of images. The most important option is the threshold option, which indicates the threshold amount of differences allowed between two images.

A full example of a test case using Visual Comparison Testing :

// example.spec.js
const { test, expect } = require('@playwright/test');

test('example test', async ({ page }) => {
  await page.goto('https://testingbot.com');
  expect(await page.screenshot('.hero')).toMatchSnapshot('hero.png');
});

Comparing binary or text formats

Next to visual comparison of images, Playwright Test is also capable of comparing texts and binary outputs.

Let's say for example that you want to make sure your DOM element keeps using the same text content. A simple way to do this is to snapshot the contents of the DOM element and compare it in later runs with the golden snapshot:

// example.spec.js
const { test, expect } = require('@playwright/test');

test('example test', async ({ page }) => {
  await page.goto('https://testingbot.com');
  expect(await page.textContent('.login')).toMatchSnapshot('login.txt');
});

Conclusion

Playwright Test gives you a basic, but powerful way to (visually) compare webpages. Give it a try - more information available on the Playwright Test Documentation page.

Setting up your own Mobile Device Lab

Jochen — Sat, 31 Aug 2019 09:31:10 +0000

As a developer, designer or QA tester, we want to make sure the app or website that we are working on, looks and behaves flawlessly across all our users' devices and browsers.

While in the past, it was mostly necessary to do cross browser testing - nowadays we also have to make sure our website works on all the different mobile devices.

During development of your website or mobile app, you can easily verify and test your app on a mobile simulator (iOS) or emulator. Overall, these tools do a great job in making sure everything works as it should.
However, simulator or emulator testing alone isn't enough, you need to make sure your app works on devices that your users carry with them in their pocket.

This article describes how to set up a Mobile Device Lab.

What is a Mobile Device Lab
A Mobile Device Lab is usually a closet or table, filled with the mobile devices you want to test on.
In its most basic form, it's a closet where devices are scattered on the shelves, each connected to a power outlet and a wifi station.

If you intend to only do manual testing, then this kind of setup is usually sufficient.
However, some tweaks, which we'll describe further in this article, might also be helpful.

If you're looking to do automated testing, for example with Selenium and Appium, then please follow along.

Getting the devices
We recommend getting mobile devices from a trusted store selling second hand/refurbished devices.
Since these are only test devices, it doesn't really matter if the back of the phone has a scratch. In case of automated testing, we don't even care if the screen has a scratch, since we'll only look at the recorded video of the automated test, and not at the actual device screen.

Power Management
This is where things get tricky; mobile devices run on batteries. If you intend to store a lot of devices in a rack solution, in some datacenter, then you need to take some things into account:

Batteries tend to overheat
Batteries degrade

Make sure you put devices on a metal shelve, since these devices get hot over time because of the battery and charging. The metal shelve will disperse heat better. Leave gaps between the devices, to make sure cool air flows between devices.

Over time, batteries degrade. A good solution for this is to invest in a good USB Hub. In a Mobile Device Lab we built, we're using a USB Hub from Cambrionix which automatically syncs and charges batteries according to the power profile of the device. This means devices will only charge when necessary, while still being synced to the host computer.

Monitor the battery levels of your devices. This is possible through ADB for Android or Instruments for iOS. Feed these into a monitoring solution to receive alerts when a battery goes bad.

Another good tip is to make sure the screen brightness of your devices is turned to the lowest level. This saves power and battery usage. Same goes for locking the screen; after each automated test, make sure to lock the screen so that the display is turned off. Leaving the display on will only draw more power and degrade batteries faster.

Connecting devices to a host computer
If you want do Automated testing, you'll need a computer that transmits the commands to the device. For iOS devices, this needs to be an Apple computer. We recommend a Mac Mini for this, it is cheap compared to other offerings, and has enough CPU and memory to handle a couple of iPhones or iPads.

To test apps on your iOS device, you'll need to install a provisioning profile connected with your host computer. This is a certificate from Apple that indicates your computer is allowed to install mobile apps on the device.

For Android testing, a cheap server can be used running Linux, for example an Intel NUC7i5. Make sure you install the latest Android SDK, authorise USB debugging between the device and the host computer, and you're all set to start automated testing.

Running tests
This part is pretty easy; there's a very good open-source project available that does most of the heavy lifting, called Appium.

Simply install Appium on the host computer and after some minimal configuration steps, you will be able to point your tests to this Appium server.

Once you start a test, you will start seeing activity on the mobile device. This is Appium controlling the device.
After the test is complete, Appium can save a screen recording of the test on your computer.
This way you can see what the test actually did on the device.

Mobile Device Cloud
If you're not looking to invest time and money in setting up a Mobile Device Lab, there's an online service offering mobile device testing. Simply point the tests you've created to this service and indicate on which physical mobile devices you want to run your tests on.
After a test completes, you'll get access to useful information such as a screen recording, log files and performance metrics.