Forem: Jonas Menesklou

AskUI Best Practices

Jonas Menesklou — Thu, 15 Sep 2022 10:52:23 +0000

Let’s be honest: Testing is a lot of fun. Not only is it extremely important for the success of your team, but it is also highly satisfying when you’ve finally found and tracked a pesky bug all the way to its original source.

However, setting up your first test case in a new environment can be tiresome, especially when every testable element has to be individually defined first. Before you know it, you are knee-deep in someone else's messy code, trying to put IDs in the right places so that you can run a simple workflow, that would take a manual tester five seconds to complete. This approach seems a little impractical, doesn’t it?

The New Way: Selector-Free Test Automation

If you feel like current testing tools are holding back your productivity - don’t worry, you are not alone. At askui, we firmly challenge the idea of selector-based testing. We believe that humans are still the best testers, because of two simple reasons:

They are quick
They are resistant to changes in the codebase (lol, obviously)

In order to automate this boring, repetitive work, future testing methods will need to be even better than humans. That’s why our mission is to provide human-level test quality at scale and at a fraction of the cost.

askui uses artificial intelligence to detect on-screen elements such as buttons and text fields automatically, allowing you to skip the usual lengthy setup process, so that you can start writing test cases right away.

How Does This Work?

The cool thing about askui is that you can approach writing test automation scripts as if you would manually perform the actions yourself. As a result, you get a more intuitive workflow that can save you a lot of time.

Before you continue: If you haven’t yet downloaded askui, follow this guide on installing the app on your machine

Setting Up a Test Suite

💭 Let’s say we’re on google.com and we want to search for an image of a cat and then download it to our computer.

First, we break this task down into steps that a user would take. Then we can recreate those steps in code.

go to google images
type “cat” in the search bar
select image
right-click + save the image

Then we begin creating our test suite, by creating two separate test blocks:

The first one is used to get an annotated screenshot, where all of the on-screen elements are enclosed within annotated bounding boxes. This will help us select the correct elements in our test case.
```
it('annotate', async () => {
    await aui.annotate()
  });
```

the second test block contains our actual test case.

xit('should click download cat image', async () => {
    await aui;
  });

You’ll notice, that the it function is “x’ed out”, which means, that the test block will be ignored, when we run the script. This is fine for now, because it does not contain anything yet.

Next, you’ll run the script to create an annotated screenshot. If you’re using VS Code, it will appear in your file navigation bar on the left.

📋 The annotations are basically the substitute for IDs in selector-based testing.
You can can click on them to copy them into your clipboard.

Writing and Debugging a Test Case

Now we can start to write our test case, by locating the elements and then executing an action on them. Remember the steps, that we wanted to recreate?

go to google images

type “cat” in the search bar

select image

right-click + save the image

In the end, your test case could look something like this.

import { aui } from './helper/jest.setup';

describe('jest with askui', () => {
  it('should click on text', async () => {
    await aui
      .click()
      .text().withText("Images")
      .rightOf()
      .text().withText("Gmail")
      .exec();
    await aui
      .typeIn("cat")
      .textfield()
      .below()
      .logo().withText("G00g.e")
      .exec()
    await aui
      .pressKey('enter')
      .exec()
    await aui
            .moveMouseTo()
            .image()
            .above()
            .text()
            .withText("pet guru Yuki Hattori explaiinICats")
            .exec()
    await aui
            .mouseRightClick()
            .exec()
    await aui
            .click()
            .text()
            .withText('save image as')
            .exec()
    await aui
            .click()
            .button()
            .withText("Save")
            .exec()
  });
  xit('annotate', async () => {
    await aui.annotate()
  });
});

Debugging

It’s possible that you’ll run into problems with locating functions. For example, when creating this tutorial, we first tried to locate the image nearest to the image title, like this

await aui
            .moveMouseTo()
            .image()
            .nearestTo()
            .text()
            .withText("pet guru Yuki Hattori explaiinICats")
            .exec()

But it turns out, that the AI uses a different metric for measuring distance between elements, which is why our script failed the first time. Then we substituted this function for above(), which fixed the problem for us.

If you have a similar issue, try to play around with the functions and see if you can tackle the problem from a different angle.

if you have a recurring or persisting issue, don’t hesitate to ask the community for help. You can be sure that your questions will be answered there. We’re excited to hear about how you apply askui on your projects.

If you have any feature requests, please feel free to post them in our featurebase board.

Best regards and happy testing!

UI Automation for Flutter Apps

Jonas Menesklou — Mon, 01 Aug 2022 18:26:00 +0000

In today's tutorial we will show how to automate Flutter applications with askui. Please also find a video tutorial here.

We assume that askui is already installed in a clean state, see here

What is Flutter?

Flutter is an open-source UI development kit from Google available since 2017. The special feature of Flutter is the possibility of cross-platform development of applications for Android, iOS, Linux, macOS, Windows, Google Fuchsia and web.

Due to this advantage, it quickly became known as a popular framework and various companies are already using it.

If you follow the interest over time, it quickly becomes clear that Flutter will probably be one of the most popular frameworks in the future.

Setting up the Flutter App

Since this tutorial is about automating flutter apps and not about development, we use the Health.ai Flutter Demo App as from Flutterflow.io as an example. The startup from Silicon Valley offers a low-code builder for creating Flutter Apps.

In principle, however, it does not matter how the Flutter app is developed and hosted. Since askui only accesses the operating system and not the application itself, we can implement a true black box test.
If you want to test a mobile application, you start it on an emulator on the test system. As soon as the emulator is visually visible, it can be controlled with askui.

Tip: Using the adb shell, the askui runner can also be installed on Android and thus run on a real device. More about this in a later tutorial.

The Health.ai app under test looks like this:

As a simple test, we would like to test the login process.

Automating with askui

The login process basically consists of the following three steps:

Entering the username, in our case an account has already been created at "info@askui.com",
Entering the password, in this case "test123",
Click on the login button.

Besides that, it matters how and the app is hosted. In our case in a web browser, so an open browser is the precondition for running the test. We can start the app using the following code in the my-first-askui-test-suite.test.ts file:



await aui.typeIn("https://app.flutterflow.io/run/1E6pv4HAPn7zBGaHYtMx").url().exec();
await aui.pressKey('enter').exec();

After running, the browser should look something like this:

If a mobile emulator is used, the app must be launched in it beforehand and the emulator must be visually visible on the screen.

Next we want to automate the filling of the text fields. To better understand how these elements are recognized, we add a help test case, which saves the screenshots with the recognized elements in a separate folder, we add the following lines:



  it('Annotation', async () => {
    await aui.annotate();
  });

With each new test run, a new html document is now stored in the report folder. If we copy the relative path of this file into our browser, we can check what the askui AI has recognized.

Now, to fill in the actual text fields, we use the following code after starting the application:



await aui.typeIn("info@askui.com").textfield().contains().withText("Email Address").exec();
await aui.typeIn("test123").textfield().contains().withText("Password").exec();

The following command is used to perform the login:



await aui.click().text().withText("Login").exec();

The subsequent verification of the successful login is checked by the appearance of the Good Morning text, which appears after the successful login:



await aui.expect().text().withText("Good Morning").exists();

Adding all up

The complete test file should now look like this:



import { aui } from './helper/jest.setup';

describe('jest with askui', () => {
  it('Flutter App Login Form', async () => {

//Open Flutter App
await aui.typeIn("https://app.flutterflow.io/run/1E6pv4HAPn7zBGaHYtMx").url().exec();
await aui.pressKey('enter').exec();

//Fill In Form
await aui.typeIn("info@askui.com").textfield().contains().withText("Email Address").exec();
await aui.typeIn("test123").textfield().contains().withText("Password").exec();

//Login
await aui.click().text().withText("Login").exec();

//Verify Login
await aui.expect().text().withText("Good Morning").exists();

  });

  it('Annotation', async () => {
    await aui.annotate();
  });

});

You can now run the whole file with the command npx jest --config ./test/jest.config.ts;. Just make sure the app is visible on the test system (in this case your screen).

You can also find the complete project here on Github.

Happy testing!

Getting Started With askui

Jonas Menesklou — Thu, 09 Jun 2022 16:06:10 +0000

Tip: Watch our how to get started video here

Test automation is still considered a bottleneck in modern software development. Especially UI automation can be quite challenging in certain cases. Despite countless frameworks and tools, it is very difficult to automate applications across operating systems, although cross-platform workflows play an increasingly important role. For example, testing a 2-factor authentication of a web application via smartphone is almost impossible to automate as an end-to-end process.

askui tries to close this gap by rethinking UI automation.

About askui

Basically, askui is a platform-independent, selector-free UI automation framework. Instead of controlling elements via selectors such as XPath or CSS selectors and thus being limited to web applications, askui controls the UI at the operating system level.

For making this possible, a neural network was trained on the appearance of UI elements in order to localize UI elements based on screenshots. These are then matched with an instruction that is unique for the respective element. The final execution of the instruction happens via mouse and keyboard control on the operating system. Such an instruction could be for example:

aui.click().button().below().text().withText("Password").exec();

If we take a look at the corresponding UI, it quickly becomes clear which action is to be executed:

The instruction describes a click on a button below a certain text, in this case Password. Visually, there is only one button here, so when the action is executed, a click on the blue Login button is performed.

Why askui

In principle, one can of course ask whether another test automation tool is needed after Selenium and co. If you look at the development of the last few years, you will notice that websites and apps are using more and more new technologies, e.g. iFrame or ShadowDOM, and third-party systems, which make it more difficult to automate them. Instead of developing a separate automation solution for each new technology, askui wants to act independently of these developments and act as a future-proof framework for UI automation of any operating system. The following advantages result from the approach:

Test instructions are written in UI language and are thus easy to understand.
No access to code selectors is needed. Thus the tests are independent of changes in the program code and run very stable.
Elements are found visually by an AI and not by pixel-matching. This makes the automation completely independent of the resolution and design of the UI elements.

In the following, we will take a look at the first steps in askui.

Setting up the IDE

askui is an open source framework with a public documentation. You can install askui in an IDE of your choice, we will use Visual Studio Code in the following. Firstly, open the folder where we want to install askui. In my example this folder is called "Installation". It should look something like this:

Next, we need to set up the npm project. For this we open the terminal and type the following command:

npm init -y

This will create a package.json file with descriptions and dependencies of the project. The next step is the installation of askui itself.

Installing askui

The following command is used for installing askui:

npm i -D askui

Thereby askui library provides everything needed for the automation of the operating system. It does not yet provide everything you need for writing and executing a test. You also need a way of

writing up the actual test,
writing up assertions to test wehther an expectation holds true and, last but not least,
a way to execute the tests, i.e., a test runner.

One framework which provides all of this out of the box is Jest which we are going to use in this example as it is quite easy to get started with and well-known. But feel free to use another test framework, such as Jasmine or Mocha. How you use the askui library should be pretty much the same across these frameworks. For installing Jest, type the following:

npm i -D jest

Furthermore, we are going to use TypeScript for writing the test instead of plain JavaScript. Run the following command to install Typescript, TS-Node for using Typescript together with Node.js and the types of Jest and Node.js.

npm i -D @types/jest ts-jest ts-node typescript

Your IDE then should look similar to this:

Now, we are ready to write our first test.

Running Your First Test

For creating your first test suite, type following in the terminal:

npx askui init

This will create a few files, read here if you want to learn more about them.

We will just focus on test/my-first-askui-test-suite.test.ts for now. This file includes your first example test which should look like this:

This test is going to click on a random text on your screen after executing. But before we execute it, we want to make sure it will execute on the right screen. Therefore, we will check the file test/helper/jest.setup.ts. There you will find a setting for display which is by default set to 0. This means the test will be executed on your main display. If you don't have any external screen this setting can be ignored. If you do have external screens, you should set it to the screen you want to automate on, e.g. for the first external monitor, change it to 1 and so on.

Now we are ready to execute our first test. Therefore, we type following command in the terminal:

npx jest test/my-first-askui-test-suite.test.ts --config ./test/jest.config.ts

You should now see the test suite being executed inside the shell and, actually, your cursor should move to some text shown on your screen and click on that text. 🎉 Congratulations! You just executed your first test suite using askui.

Tip: If you want to see what the AI model detects on your screen, use the annotateInteractively (see here) command. It will create an overlay where you can check all detected UI elements and their classification.