Forem: ryanrosello-og

Model based testing using Playwright and xstate, what I've learnt so far

ryanrosello-og — Fri, 15 Jul 2022 11:49:29 +0000

This article will guide you through the very basic setup of integrating model based testing using xstate combined with Playwright. Hopefully, you should already have a good grasp of the key concepts related to xstate and its capabilities. If not, don't panic. I suggest you view this short YT Write fewer tests! Model-based testing in React — David Khourshid

The test model

In our example, we will generate a test model to represent a user searching for documents on Playwright.dev website

The two basic paths through this process:

Perform a search that yeilds results
Perform a search that does not yeild any results

Setup

To incorporate xstate into your test project requires a few simple steps:

Install xstate / xstate-test

npm install @xstate/test xstate

Import the necessary xstate libraries into your spec file:

import { createMachine } from "xstate";
import { createModel } from "@xstate/test";

Create the state machine

const documentationSearchMachine = createMachine({
  id: 'documentSearch',
  initial: 'idle',
  states: {
    idle: {
      on: {
        PERFORM_SEARCH: 'resultsFound',
        PERFORM_INVALID_SEARCH: 'noResults',
      },
    },
    resultsFound: {
      on: {
        RESET_SEARCH: 'readyForNextSearch',
      },
    },
    noResults: {
      on: {RESET_SEARCH: 'readyForNextSearch'},
    },
    readyForNextSearch: {
      type: 'final',
    },
  },
});

ℹ️ You can use the xstate visualizer to preview the machine above: Go to stately visualizer and paste the snippet above.

Create the model

These are the "actions" that does the heavy lifting of transitioning one state to another.

  const testModel = createModel(documentationSearchMachine, {
    events: {
      PERFORM_SEARCH: async () => {
        const validSearchTerm = 'Text selector';
        await page.locator('[aria-label="Search"]').click();
        await page.locator('#docsearch-input').type(validSearchTerm);
      },
      PERFORM_INVALID_SEARCH: async () => {
        const invalidSearchTerm = 'pztr';
        await page.locator('[aria-label="Search"]').click();
        await page.locator('#docsearch-input').type(invalidSearchTerm, {delay: 300});
      },
      RESET_SEARCH: async () => {
        await page.goto('https://playwright.dev/');
      },
    },
  });

Add your tests

Creating the tests is super easy since we let xstate generate our test plans for us. The snippet below basically generates the tests dynamically based on the model.

  test.beforeAll(async ({browser}) => {
    page = await browser.newPage();
  });

  test.beforeEach(async () => {
    await page.goto('https://playwright.dev/');
  });

  test.afterAll(async ({browser}) => {
    for(let context of browser.contexts())
      await context.close()
  });

  const testPlans = testModel.getSimplePathPlans();
  testPlans.forEach((plan, i) => {
    plan.paths.forEach((path, i) => {
      test(path.description, async ({}) => {
        await path.test(page);
      });
    });
  });

Run the tests

The most important line of the code above is the const testPlans = testModel.getShortestPathPlans();, when this method is utilized to generate the paths, it yields the following results:

As the name suggest, this executes the most basic paths through the machine.

To enable a more exhaustive set of tests, you should use:

const testPlans = testModel.getSimplePathPlans();

This yields more paths through model, as shown in the results below:

☠️ Warning! This method of execution will be slow, especially when your model involves time intensive workflows.

I would suggest you split your execution into two:

getShortestPathPlans - iterate through the basic paths through your model for quicker execution as part of your CI
Execute the full breadth of your paths out of hours/overnight using getSimplePathPlans

Refine the machine to include assertions

Refer the new keys added below (meta) properties. This specifies an assertion to be performed to ensure our model successfully transitioned from one state to another.

  const documentationSearchMachine = createMachine({
    id: 'documentSearch',
    initial: 'idle',
    states: {
      idle: {
        on: {
          PERFORM_SEARCH: 'resultsFound',
          PERFORM_INVALID_SEARCH: 'noResults',
        },
        meta: {
          test: async () => {
            await expect(page.locator('.navbar__title')).toHaveText('Playwright');
          },
        },
      },
      resultsFound: {
        on: {
          RESET_SEARCH: 'readyForNextSearch',
        },
        meta: {
          test: async () => {
            await expect(
              page.locator('#docsearch-list .DocSearch-Hit-title >> text=Selecting elements by label text'),
              'correct search result found'
            ).toBeVisible();
          },
        },
      },
      noResults: {
        on: {RESET_SEARCH: 'readyForNextSearch'},
        meta: {
          test: async () => {
            await expect(
              page.locator('.DocSearch-NoResults'),
              'no results panel should be displayed'
            ).toBeVisible();
          },
        },
      },
      readyForNextSearch: {
        type: 'final',
        meta: {
          test: async () => {
            await expect(page.locator('.navbar__title')).toHaveText('Playwright');
          },
        },
      },
    },
  });

Initial concerns

I still have some lingering doubts about using this approach for model-based testing but I don't see any better alternatives. Some of my outstanding concerns are outlined below but it will be addressed the more I use the tool

How much maintenance overhead will this introduce? - TBD
What happens if the model changes, e.g. a new state added/deleted - TBD
How do I debug/troubleshoot issues?

Anything data driven is notoriously difficult to troubleshoot, luckily we can use vscode's ability to specify conditional breakpoints.

Alternatively, you can drop in a test.skip statement to isolate only certain paths that you would like to execute

e.g.

      test(path.description, async ({}) => {
        test.skip(path.description!=='via PERFORM_SEARCH → RESET_SEARCH')
        await path.test(page);
      });

Takeaways

Is this my life now? Do we model all test cases from now on? If I was to hire a backpacker to maintain/extend the test code base, how much upskilling will be involved?

The key advantages to using xstate is the ability to test all the different paths through an application, traditional test automation approaches handled this poorly.

Creating a test model can be difficult, especially for complex end to end scenarios. Some key learnings:

Keep your model simple, if you look through the some of the examples xstate models. You will notice that it can get hairy pretty quickly
Use the model visualizer
Design and validate your model, preferably have someone else review it
Iterate. Design and validate and refine your model, preferably have someone else review it. ^ Do these before hacking away
Use this as a collaborative tool to facilitate a better model

This article barely scratched the surface of the full capabilities of model-based testing using xstate. I'll leave it upto you to explore and apply it to your solution.

The final working solution:

Validate complicated graphics rich pdf documents using Playwright

ryanrosello-og — Sun, 10 Jul 2022 10:32:52 +0000

This article will take your pdf verification skills to the next level using Playwright.

The pdf document we will use for this example will be the "Tesla Powerwall 2 Datasheet". This file is hosted at following location https://oedtrngbj.wpengine.com/wp-content/uploads/Powerwall_2_AC_Datasheet_EN_NA.pdf

Observe that this pdf document contains multiple pages, some illustrations and a table.

Caveats

Let me be upfront with the following issues I have encountered with this approach:

This solution only seems to work when the test is run using chromium in headed mode
The elements contained within pdf viewer component are not accessible to Playwright, this means you will not be able to mask/hide dynamic elements in the pdf. For example, customer id or dynamic date/time stamps
We are using Playwrights' built in visual comparison library. It is advisable that you get familiar with the maintenance required to keep the baseline images up to date. See the Visual comparisons page on the Playwright documentation

If you are happy with these compromises, read on!

page.setContent() + toMatchSnapshot() = 🤩

Using the setContent(), load the pdf into an iframe like so:

  const pdfResource =
    'https://oedtrngbj.wpengine.com/wp-content/uploads/Powerwall_2_AC_Datasheet_EN_NA.pdf';
  let iframe = `<iframe src="${pdfResource}#zoom=60" style="width: 100%;height:100%;border: none;"></iframe>`;
  await page.setContent(iframe);
  await page.waitForTimeout(5000);

NOTE: You may need to experiment with the zoom level, width, height attributes to suit your needs

ANOTHER NOTE: The waitForTimeout function is used here to wait for the pdf contents to be loaded into the iframe

We will make use of Playwrights' assertion expect(screenshot).toMatchSnapshot(name[, options]) => https://playwright.dev/docs/test-assertions#screenshot-assertions-to-match-snapshot-1, to capture a screenshot of a particular element matching a locator, in our case - we will need to take a screenshot of the iframe above with the PDF file fully loaded to particular page.

Our solution will make use of this function:

  expect(await page.locator('iframe').screenshot()).toMatchSnapshot();

The completed test will look like this ...

import { test, expect } from '@playwright/test';

test('validate a complex pdf', async ({ page }) => {
  const pdfResource =
    'https://oedtrngbj.wpengine.com/wp-content/uploads/Powerwall_2_AC_Datasheet_EN_NA.pdf';
  let iframe = `<iframe src="${pdfResource}#zoom=60" style="width: 100%;height:100%;border: none;"></iframe>`;
  await page.setContent(iframe);
  await page.waitForTimeout(5000);
  expect(await page.locator('iframe').screenshot()).toMatchSnapshot();
});

Run the test. It should fail with the following error.

Playwright has not found a golden snapshot of the element and hence on the very first test execution, it will automatically generate this file for you. You will need to commit these files into your repo.

Rerun the test again, this time it should pass since it will already have a baseline image to compare against.

Ok, that's nice. We managed to validate the first page of the pdf.

But most pdfs you will encounter out in the wild will contain multiple pages. Let us ammend our test to cater for multiple pages.

test('validate a complex pdf II, all pages', async ({page}) => {
  const numberOfPages = 2;
  for (let i = 1; i < numberOfPages + 1; i += 1) {
    let pdfResource =
      'https://oedtrngbj.wpengine.com/wp-content/uploads/Powerwall_2_AC_Datasheet_EN_NA.pdf';
    let iframe = `<iframe src="${pdfResource}#zoom=60&page=${i}" style="width: 100%;height:100%;border: none;"></iframe>`;
    await page.setContent(iframe);
    await page.waitForTimeout(5000);
    expect(await page.locator('iframe').screenshot()).toMatchSnapshot({
      name: `pdf_validation_page_${i}.png`,
    });
  }
});

Job done.

Other improvements for you to consider

(all totally optional and it entirely up to you to implement)

Dynamically determine the number of pages, the example above uses a predefined value for the expected number of pages.
Remove the hard coded waitForTimeout and implement a better way of waiting for the contents to be loaded.

Final solution

Easiest way to setup a VSCode launch config for your Playwright tests

ryanrosello-og — Sat, 09 Jul 2022 11:16:20 +0000

In this article we will look at an alternative approach to the Playwright VSCode extension by configuring our own VSCode launch config file. This will allow us to have finer control over specific launch configurations settings for our tests.

To begin with, create a new script in your package.json file called vscode-debug. (You might need to tweak the name of the playwright config file to suite your needs.)



"scripts": {
  "vscode-debug": "playwright test --config ./playwright.config.ts --workers=1"
},

Next, create a .vscode folder at the root of your project. In this folder, create a new json file named launch.json.

Input the following placeholder configuration details into the file



{
  "version": "0.2.0",
  "configurations": [
    {
      "type": "node",
      "request": "launch",
      "name": "Playwright tests",
      "console": "integratedTerminal",
      "skipFiles": [
        "<node_internals>/**"
      ],

    }
  ]
}

If you try and run the profile via clicking the play button below (or pressing F5), you will notice that it will do nothing.

Let us make configuration usable by adding the following entries:



"runtimeExecutable": "npm",
"runtimeArgs": ["run-script", "vscode-debug", "--", "${fileBasename}"],

Setting the runtimeExecutable to npm will allow you to invoke the npm script (vscode-debug) you created above. The --, "${fileBasename}" arguments will tell the launcher to pass the current highlighted file you have opened in your editor.

We can now test our settings by opening a spec file into your editor and clicking the run button:

To debug, simply place a breakpoint next to the line of code you desire and click the play button:

If you were to highlight and set focus onto a file that is not a spec file, running the profile should result in the following:

Ok so far so good, what if we wanted to launch into the Playwright inspector? This can be achieved by simply duplicating the profile created above and tweaking it slightly by changing the name to Playwright Inspector and we introduce a new key env which sets the PWDDEBUG environment variable to 1:



{
  "type": "node",
  "request": "launch",
  "name": "Playwright Inspector",
  "console": "integratedTerminal",
  "skipFiles": [
    "<node_internals>/**"
  ],
  "runtimeExecutable": "npm",
  "runtimeArgs": ["run-script", "vscode-debug", "--", "${fileBasename}"],
  "env": {
    "PWDEBUG": "1",
  }
}

Test your new profile by selecting "Playwright Inspector" and clicking the play button.

Pretty easy. You can get creative with these profiles to suit your needs. Other use cases could include:

Setting up profile to launch your tests to execute on various test environments
Create a launch profile to execute a test against a specific browser.

The possibilities are endless 🤯

Final solution

Verify PDF contents using Playwright and pdf2json

ryanrosello-og — Fri, 08 Jul 2022 10:49:11 +0000

In this tutorial we will use Playwright inconjuction with pdf2json to validate contents of a pdf file. This is very common task that you will normally encounter when creating end to end automated tests.

The pdf file we will use for this example is plain old textual based pdf containing 6 pages. For simplicity, I have stored this file pdf_sample.pdf into the root folder of the project.

Our goals are:

validate the meta informaiion (keywords:"Standard Fees and Charges, 003-750, 3-750") contained within the file
ensure the pdf file indeed has 6 pages
assert whether the PDF file contains the correct text "When we may charge fees"

First up, you will need to add pdf2json to your project using yarn (or npm):

yarn add pdf2json -D

Import pdf2json into your spec file and create the initial scaffolding for our tests:



import PDFParser from 'pdf2json';
import { test, expect } from '@playwright/test';

test.describe('assert PDF contents using Playwright', () => {
  test.beforeAll(async () => {
  })

  test('pdf file should have 6 pages', async () => {
  });

  test('contains the correct subheading text', async () => {
  });

  test('shows the correct meta information (keywords)', async () => {
  });
});

Create a simple helper function that does the heavy lifting of parsing and loading the pdf contents into a variable:



async function getPDFContents(pdfFilePath: string): Promise<any> {
  let pdfParser = new PDFParser();
  return new Promise((resolve, reject) => {
    pdfParser.on('pdfParser_dataError', (errData: {parserError: any}) =>
      reject(errData.parserError)
    );
    pdfParser.on('pdfParser_dataReady', (pdfData) => {
      resolve(pdfData);
    });

    pdfParser.loadPDF(pdfFilePath);
  });
}

Create variable called pdfContents scoped within the describe block:

let pdfContents: any

Update the beforeAll to read the contents of the pdf into the variable



  test.beforeAll(async ({}) => {
    pdfContents = await getPDFContents('./pdf_sample.pdf')
  })

If you were to debug and inspect the shape of the pdfContents you will notice that the first 2 tests are quite easy to assert.



  test('pdf file should have 6 pages', async () => {
    expect(pdfContents.Pages.length, 'The pdf should have 6 pages').toEqual(6);
  });

  test('shows the correct meta informaion (keywords)', async () => {
    expect(pdfContents.Meta.Keywords, 'PDF keyword was incorrect').toEqual('Standard Fees and Charges, 003-750, 3-750');
  });

However, the last test (assert if "When we may charge fees" is contained in the file) is a little bit more convulted. You will need to expand the Pages array and find the page where you expect the text to exists. You will then need to inspect Texts array to find the text that you are looking for. In our example it was found in first page on the fourth line. This equates to pdfContents.Pages[0].Texts[3].R[0].T

One last complication remains, the raw text that we require "When%20we%20may%20charge%20fees" seems to be encoded. We can easily strip out the encoding use the decodeURI function.



  test('contains the correct subheading text', async () => {
    const rawText = pdfContents.Pages[0].Texts[3].R[0].T
    expect(decodeURI(rawText), 'The subheading text was incorrect').toEqual('When we may charge fees');    
  });

Our final test

Conclusion

I have demonstrated how you can easily verify contents of a pdf using Playwright and pdf2json. We have worked with a very basic pdf containing textual information. Unfortunately, pdf2json may not be able to handle more complex PDF files. YMMV 🥳🚀

Use playwright to download files from remote servers

ryanrosello-og — Sun, 03 Jul 2022 07:00:54 +0000

Downloading files from a remote server is often a challenging prospect when it comes to incorporating the process into an automated test. This task can be easily performed by using Playwrights' native REST API helper library.

The example below is a typical pseudo Download button with a href pointing to location of a file on a remote server.

There are numerous ways of tackling this issue using Playwright. Our example below will utilise Playwrights' ability to perform REST calls.

Firstly, you'll need to fs



import * as fs from 'fs/promises'  // For saving the file contents to disk

Within your test, define the baseUrl and the resource path of the file for download.



const baseUrl = 'https://www.tesla.com/'
const resource = 'ns_videos/2021-tesla-impact-report.pdf'

Simply invoke the get method from the API testing helper



const getResp = await page.context().request.get(`${baseUrl}${resource}`)

The response body will be a buffer type. We can save this raw content into a file using the standard fs library. In the example below, I opted to include a random timestamp to the downloaded file. This is entirely optional, your implementation may vary.



let responseBuffer = await getResp.body()
await fs.writeFile(`${new Date().valueOf()}_downloaded_pdf_from_tesla.pdf`, responseBuffer, 'binary')

When the test is executed, the downloaded file (pdf in this example) will appear on your local disk, like so …

Downloading the file directly into the project root folder may not be ideal in most scenarios. Let us make one more change to our test and output the file into a different directly.

Add testInfo to the test definition:



test('able to download a remote file to disk', async ({ page }, testInfo) => {

Modify the download path to use the output path provided by the testInfo object.



const fileName = `${new Date().valueOf()}_downloaded_pdf_from_tesla.pdf`
const downloadPath = testInfo.outputPath(fileName)
await fs.writeFile(downloadPath, responseBuffer, 'binary')

Rerunning the test will result in the downloaded file being persisted within the test-results folder:

Forem: ryanrosello-og

Model based testing using Playwright and xstate, what I've learnt so far

The test model

Setup

Create the model

Add your tests

Run the tests

Refine the machine to include assertions

Initial concerns

Takeaways

The final working solution:

Validate complicated graphics rich pdf documents using Playwright

Caveats

page.setContent() + toMatchSnapshot() = 🤩

Other improvements for you to consider

Final solution

Easiest way to setup a VSCode launch config for your Playwright tests

Final solution

Verify PDF contents using Playwright and pdf2json

Our goals are:

Conclusion

Use playwright to download files from remote servers

The final solution: