Forem: Mike Stop Continues

How to Run Snapshot Tests in CI/CD with Playwright

Mike Stop Continues — Thu, 07 Mar 2024 00:00:00 +0000

Visual tests are a powerful tool, but they're hell to work with in CI/CD. They're slow, flaky, a pain to debug, and difficult to maintain.

Getting them going is no easy task. After all, you need some way to store the snapshots between runs, and you need some way to let your pipeline know when it's OK to update the snapshot cache.

Thankfully, I took care of all this for you. :)
This article solves all of these problems, and lays the foundation for you to take your visual tests to the next level.

If you get value here, you'll love my Ultimate Guide to Visual Testing in Playwright. It covers advanced testing techniques, fine-tuning your snapshot configuration, and more.

Let's get started...

1. Configure Playwright for CI/CD

Your CI/CD environment is not like your local environment, and we should account for that in our Playwright configuration.

Update your playwright.config.ts with the following options. Feel free to merge these with any other options you've set so far:



import {defineConfig, devices} from '@playwright/test';

const isCI = !!process.env.CI;

export default defineConfig({
  timeout: 1000 * 60,
  workers: isCI ? 1 : '50%',
  retries: isCI ? 2 : 0,
  forbidOnly: isCI,

  outputDir: '.test/spec/output',
  snapshotPathTemplate: '.test/spec/snaps/{projectName}/{testFilePath}/{arg}{ext}',
  testMatch: '*.spec.{ts,tsx}',

  reporter: [
    ['html', {
      outputFolder: '.test/spec/results', 
      open: 'never',
    }],
    isCI ? ['github'] : ['line'],
  ],
});

Here's what's going on:

The top few options account for the limited resources available in a CI environment. Tests will run slower, and they'll be a little more flaky, so we need to adapt accordingly.
The next batch of options organize all of your tests' outputs into the .test directory. This will make it easy to cache. Add .test/ to your .gitignore file. None of this stuff should be committed to your repo.
Lastly, we're going to generate html and github reports in CI/CD. The html report will be useful for debugging, and the github report will be useful for reviewing results with minimal output.

2. Run your tests in CI/CD

Next, let's create a basic pipeline. It doesn't do everything (yet), but it's a solid foundation to work from.

Create a file at .github/workflows/visual-tests.yml:



name: Visual Tests

on:
  push:
    branches:
      - "*"

  pull_request:
    branches:
      - "*"

jobs:
  run-tests:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Set up Node
        uses: actions/setup-node@v3
        with:
          cache: pnpm
          node-version-file: .nvmrc
          check-latest: true

      - name: Install deps
        run: |
          npm install
          npx playwright install

      - name: Test
        run: npx playwright test --ignore-snapshots

Notice that we're running our tests with the --ignore-snapshots flag. Since we have no way to store snapshots between test runs, this is literally the only way to get our tests to pass.

If you push your repo to Github now, this workflow will run, and it will pass, but it won't guarantee anything about your app.

Let's remedy that...

3. Store snapshots between runs

For our visual tests to work, we'll need to store snapshots between runs. For this, we'll cache them using Github's "artifacts" feature.

Second, let's also write a step that generates new snapshots if they don't exist. This will solve the problem of our tests failing the first time they run on a branch.

Third, let's stop ignoring our snapshot assertions.

Here's the updates:



      - name: Install deps
        run: |
          npm install
          npx playwright install

      - name: Set up cache
        id: cache
        uses: actions/cache@v4
        with:
          key: cache/${{github.repository}}/${{github.ref}}
          restore-keys: cache/${{github.repository}}/refs/heads/master
          path: .test/**

      - name: Initialize snapshots
        if: ${{steps.cache.outputs.cache-hit != 'true'}}
        run: npx playwright test --update-snapshots

      - name: Test
        run: npx playwright test

Notice that our cache key includes the current branch as well as a fallback to the master branch. That way a new branch has some material to work from on the first push.

If you trigger the pipeline now, on the first run, you'll see snapshots generated, the tests pass, and the cache updated. On subsequent runs, you'll see the cache hit, no new snapshots generated, and the tests still pass.

Not bad, but what happens when the snapshots need to be updated?

4. Trigger a snapshot update

Now that we have visual tests that really work, let's change our workflow so that we can update snapshots on demand.

First, let's parameterize our workflow...



on:
  push:
    branches:
      - "*"

  pull_request:
    branches:
      - "*"

  # Allow updating snapshots during manual runs
  workflow_call:
    inputs:
      update-snapshots:
        description: "Update snapshots?"
        type: boolean

  # Allow updating snapshots during automatic runs
  workflow_dispatch: 
    inputs:
      update-snapshots:
        description: "Update snapshots?"
        type: boolean

workflow_call allows us to manually trigger the workflow with or without updating snapshots. workflow_dispatch allows other Github workflows the same set of options.

Here's what the manual widget looks like:

Let's also update "Initialize snapshots" to trigger an update when this parameter is enabled:



      - name: Initialize snapshots
        if: ${{steps.cache.outputs.cache-hit != 'true' || inputs.update-snapshots == 'true'}}
        run: npx playwright test --update-snapshots --reporter html

Lastly, let's upload our HTML report as an artifact, so that we can reference it before deciding if we should ask the workflow to update snapshots... or if we need to debug the problem:



      - name: Test
        continue-on-error: true
        run: npx playwright test

      - name: Upload test report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: playwright-report
          path: .test/spec/results/
          retention-days: 30
          overwrite: true

Notice that our "Test" step now enables continue-on-error, so that we can upload our report even if the tests fail.

The new "Upload test report" step uploads the report as an artifact. In the workflow results, you'll see a direct link to a zip file containing our report.

5. Speed up your CI/CD pipeline... a lot!

Things are going great, but there's one more thing we can do to make visual tests even better: make them blazingly fast!

You've probably noticed two steps are slowing your pipeline down:

Installing Playwright browsers.
Running tests serially.

Thankfully, we can fix both of these issues, and most-likely for free!

BrowserCat hosts a fleet of Playwright browsers in the cloud that you can connect to with just a few lines of code.

If you do so, you won't have to install browsers in your CI/CD environment, and you'll be able to run your tests entirely in parallel.

And the best part? BrowserCat has an awesome forever-free plan. Unless you're running a huge team, you'll probably never have to pay a dime.

So let's get started!

First, sign up for a free account at BrowserCat.

Second, create an API key and store it as a secret named BROWSERCAT_API_KEY in your Github repository. You can do this by navigating to your repository, clicking "Settings," then "Secrets," then "Actions," then "New repository secret."

Third, let's update playwright.config.ts to use BrowserCat:



const isCI = !!process.env.CI;
const useBC = !!process.env.BROWSERCAT_API_KEY;

export default defineConfig({
  timeout: 1000 * 60,
  workers: useBC ? 10 : isCI ? 1 : '50%',
  retries: useBC || isCI ? 2 : 0,
  maxFailures: useBC && !isCI ? 0 : 3,
  forbidOnly: isCI,

  use: {
    connectOptions: useBC ? {
      wsEndpoint: 'wss://api.browsercat.com/connect',
      headers: {'Api-Key': process.env.BROWSERCAT_API_KEY},
    },
  },
});

In the updates above, you'll notice we've increased parallelization to 10 workers when using BrowserCat. This is a good starting point, but you can increase it significantly more without issue.

Notice that this config connects to BrowserCat whenever BROWSERCAT_API_KEY is defined. This is useful for running tests locally, as well as in CI/CD.

OK, now here's the last bit of magic... Let's stop installing Playwright's browsers every time we run our pipeline:



      - name: Install deps
        run: npm install

If you run your pipeline now, you'll witness a dramatic speed-up, even with just a single test in your suite. And with 1000 free credits per month, you can run your pipeline for hours on end with plenty of time left over.

Next Steps...

You've got your CI/CD pipeline working. Now it's time to write some tests!

Be sure to check out my Ultimate Guide to Visual Testing in Playwright for advanced snapshot testing strategies and ready-made solutions for running visual tests in CI/CD.

In the meantime, happy testing!

Configuring Snapshot Tests in Playwright

Mike Stop Continues — Wed, 06 Mar 2024 00:00:00 +0000

Playwright's snapshot tests are incredibly powerful, but they can be very flaky without the right configuration.

By the end of this article, you'll have a solid understanding of how to configure your visual tests for the perfect balance of stability, security, and validity.

If you're interested in going deeper into visual testing, you'll love my Ultimate Guide to Visual Testing in Playwright. It covers advanced testing techniques, CI/CD integration, and more.

Let's get started...

Custom Snapshot File Names

Playwright automatically names your snapshots based on the name of the test. If you're only using snapshots for visual testing, that's fine.

However, many users like to repurpose these images for documentation, style guides, and deployment reports. These tasks demand consistent file names.

Name your snapshots like so:

test('custom snapshot names', async ({page}) => {
  await page.goto('https://www.browsercat.com');
  const $hero = page.locator('main > header');

  await expect(page).toHaveScreenshot('home-page.png');
  await expect($hero).toHaveScreenshot('home-hero.png');

  const $footer = page.locator('body > footer');
  const footImg = await $footer.screenshot();

  expect(footImg).toMatchSnapshot('home-foot.png')
});

Note: Custom snapshot names don't completely control the file name unless you also configure custom directories. With the default configuration, Playwright adds a suffix to your file names to ensure they're unique across projects.

Read on for how to customize the snapshot directory structure...

Custom Snapshot Directories

By default, Playwright stores snapshots in the same directory as the test file that created them. This method has numerous shortcomings:

It makes navigating your codebase more difficult.
It's hard to exclude snapshots from version control.
It's not easy to cache results between CI/CD runs.
And it gets in the way of custom snapshot file names.

So let's tell Playwright to store our snapshots in a custom directory. Update your playwright.config.ts:

export default defineConfig({
  snapshotPathTemplate: '.test/snaps/{projectName}/{testFilePath}/{arg}{ext}',

  // etc.
});

With the above configuration, if our test file is stored at ./tests/homepage.spec.ts, the Playwright will store our snapshots like so:

.test/
  snaps/
    tests/
      homepage.spec.ts/
        home-page.png
        home-hero.png
        home-foot.png

With this pattern, you can exclude your snapshots from version control, you can cache your results in CI/CD, and you can easily review failed tests as they arise.

Read more about the your options in the Playwright docs for snapshot path templates.

Make Visual Tests More Forgiving

Out of the box, visual tests are very strict. If a single pixel fails, your test fails. Thankfully, Playwright provides numerous controls for tuning how sensitive your visual tests should be.

Here are your options:

threshold: How much must a single pixel vary for it to be considered different. Values are a percentage from 0 to 1, with 0.2 as the default.
maxDiffPixels: The maximum number of pixels that can differ while still passing the test. By default, this option is disabled.
maxDiffPixelRatio: The maximum percentage of pixels that can differ while still passing the test. Values are a percentage from 0 to 1, but this control is disabled by default.

You can tune these options globally or per-assertion.

Configure forgiveness globally

To configure your snapshot options globally, update your playwright.config.ts:

export default defineConfig({
  expect: {
    toHaveScreenshot: {
      threshold: 0.25,
      maxDiffPixelRatio: 0.025,
      maxDiffPixels: 25,
    },
    toMatchSnapshot: {
      threshold: 0.25,
      maxDiffPixelRatio: 0.025,
      maxDiffPixels: 25,
    }
  },
});

Override forgiveness per-assertion

To override your global snapshot options, update your test file:

test('forgiving snapshots', async ({page}) => {
  await page.goto('https://www.browsercat.com');
  const $hero = page.locator('main > header');

  await expect(page).toHaveScreenshot({
    maxDiffPixelRatio: 0.15,
  });

  await expect($hero).toHaveScreenshot({
    maxDiffPixels: 100,
  });
});

How do I tune these options?

To converge on the correct setting for your app, review the failing "diff" images produced by your tests. In those images, yellow pixels indicate pixels that are within the threshold allowance, while red pixels fail the test.

First try to increase threshold, and see if you can gobble up the red pixels in your snapshots. But be careful: a high threshold will cause false negatives. I wouldn't go any higher than 0.35, and that's high.

If threshold doesn't work, focus on maxDiffPixelRatio and maxDiffPixels. And consider the trade-offs these options entail.

Since maxDiffPixelRatio is relative to image size, it's more likely to yeild good results across a wide range of images. This makes it a good contender for a global setting... but only if you set it conservatively! After all, 10% of a full page image will allow a lot of variation.

On the other hand, since maxDiffPixels is a constant value, it gives you much better control on individual tests. But it's also much more risky if applied globally, even at a low setting. For small images, a high maxDiffPixels might constitute a huge percentage of the image.

Next Steps...

That's really all there is to configuring your visual tests, but there's so much more to learn. For advance snapshot testing strategies and ready-made solution for running visual tests in CI/CD, check out my Ultimate Guide to Visual Testing in Playwright.

In the meantime, happy testing!

Advanced Snapshot Testing in Playwright

Mike Stop Continues — Tue, 05 Mar 2024 00:00:00 +0000

Playwright's snapshot assertions are an incredibly powerful tool for ensuring your app's UI remains consistent across code changes, browsers, and devices. But they're not always easy to use.

This article dives deep on snapshot testing in Playwright, covering a wide range of features and techniques. By the end, you'll be a snapshot testing master, ensuring your app's flawless visual consistency across browsers and devices.

When you're finished, check out my Ultimate Guide to Visual Testing with Playwright. It covers setup, advanced configuration, and running your visual tests in CI/CD.

Let's go!

Page vs. Element Snapshots

Playwright's visual testing API allows you to take snapshots of the entire page or just a specific element.

But when is the right time to use one over the other?

When should I use page snapshots?

Page snapshots are excellent for verifying the entire page works as expected. Use page snapshots to test layout, responsiveness, and accessibility.

But be warned: Page snapshots can be flaky. After all, if anything within the viewport changes, the entire snapshot will fail. We'll cover strategies for minimizing these effects later, but for now, it's wise to consider page snapshots a powerful, blunt instrument.

You've already seen a page snapshot in action. Here's a refresher:

test('page snapshot', async ({page}) => {
  await page.goto('https://www.browsercat.com');
  await expect(page).toHaveScreenshot();
});

When should I use element snapshots?

Element snapshots, as you expect, focus exclusively on a single page element. This makes them an excellent choice for testing components in isolation, or for verifying an element behaves as expected within a certain context.

Element snapshots are substantially less brittle than page snapshots, but they require a bit more overhead to set up. After all, their narrow targeting means you need more of them to cover the same surface area as a page snapshot.

Here's an example of an element snapshot:

test('element snapshot', async ({page}) => {
  await page.goto('https://www.browsercat.com');
  const $button = page.locator('button').first();
  await expect($button).toHaveScreenshot();
});

Working with Page Snapshots

Let's explore some useful features and common use-cases for page snapshots...

Cropping Page Snapshots

Sometimes the entire viewport isn't necessary to prove your test passes. And sometimes a portion of the viewport changes frequently by design, turning an otherwise great test into a flake.

In these cases, it's best to crop your snapshot to the area of interest. Here's an example:

test('cropped snapshot', async ({page}) => {
  await page.goto('https://www.browsercat.com');
  const {width, height} = page.viewportSize();

  await expect(page).toHaveScreenshot({
    // square at the center of the page
    clip: {
      x: (width - 400) / 2, 
      y: (height - 400) / 2, 
      width: 400, 
      height: 400,
    },
  });

  await expect(page).toHaveScreenshot({
    // top slice, maximum possible width
    clip: {x: 0, y: 0, width: Infinity, height: 16},
  });
});

Snapshot the Entire Page

By default, Playwright takes a snapshot of the current viewport. This is typically what you want, as the larger the snapshot is, the more likely it is for your test to fail.

However, full page snapshots have their place. For example, if you're testing that a page looks the same across different browsers, the the easiest solution is to snapshot the entire page. And since for this kind of test, you aren't storing the snapshot from previous runs, you're not going to end up with an overly flaky test.

Here's how you take a full page snapshot:

test('full page snapshot', async ({page}) => {
  await page.goto('https://www.browsercat.com');
  await expect(page).toHaveScreenshot({
    fullPage: true,
  });
});

Scroll Before Taking a Page Snapshot

When working with page snapshots, you'll often want to scroll the page before the visual assertion.

Here's how:

test('scroll before snapshot', async ({page}) => {
  await page.goto('https://www.browsercat.com');

  await page.evaluate(() => {
    document
      .querySelector('#your-element')
      ?.scrollIntoView({behavior: 'instant'});
  });

  await expect(page).toHaveScreenshot();
});

Note: While Playwright has a .scrollIntoViewIfNeeded() method, it will not scroll the element to the top of the viewport. So I recommend the solution above. It will make full use of your viewport and ensure your snapshot is consistent between runs.

Working with Element Snapshots

Element snapshots are much more "in the weeds" than page snapshots. They bring a lot of power and flexibility.

Let's explore some examples...

Test Element Interactivity

As your component library grows, it becomes harder and harder to keep track of every state of every element in your library.

In the following example, we snapshot a form input across various states:

test('element states', async ({page}) => {
  await page.goto('https://www.browsercat.com/contact');
  const $textarea = page.locator('textarea').first();

  await expect($textarea).toHaveScreenshot();
  await $textarea.hover();
  await expect($textarea).toHaveScreenshot();
  await $textarea.focus();
  await expect($textarea).toHaveScreenshot();
  await $textarea.fill('Hey, cool cat!');
  await expect($textarea).toHaveScreenshot();
});

Test Element Responsiveness

When working with responsive designs, it's important to ensure your elements look good across the full range of screen sizes.

Use element snapshots to ensure your components look good at various breakpoints.

test('element responsiveness', async ({page}) => {
  const viewportWidths = [960, 760, 480];
  await page.goto('https://www.browsercat.com/blog');
  const $post = page.locator('main article').first();

  for (const width of viewportWidths) {
    await page.setViewportSize({width, height: 800});
    await expect($post).toHaveScreenshot(`post-${width}.png`);
  }
});

Advanced Snapshot Techniques

Page and element snapshots share many common configuration options. Let's explore the most useful among them...

Masking Portions of a Snapshot

Sometimes, you'll want to exclude certain portions of a snapshot. A sub-element or sub-region may change frequently, contain sensitive information, or be irrelevant to the test. For example, a timestamp, an animation, a user email address, or a rotating ad.

Playwright provides the ability to "mask" these areas, replacing them with a consistent bright color unlikely to be confused for the content of your site.

Here's an example:

test('masked snapshots', async ({page}) => {
  await page.goto('https://www.browsercat.com');
  const $hero = page.locator('main > header');
  const $footer = page.locator('body > footer');

  await expect(page).toHaveScreenshot({
    mask: [
      $hero.locator('img[src$=".svg"]'),
      $hero.locator('a[target="_blank"]'),
    ],
  });

  await expect($footer).toHaveScreenshot({
    mask: [
      $footer.locator('svg'),
    ],
  });
});

And here's what the first masked snapshot looks like:

Keeping Styles Constant During Snapshots

Visual tests are valuable because they catch unexpected changes to your app's appearance. Some page elements are too unreliable to include as-is.

Thankfully, we can include some basic CSS for the duration of a snapshot that restrains or hides troublesome elements from the page.

Here's how:

test('consistent styles', async ({page}) => {
  await page.goto('https://www.browsercat.com');
  const $hero = page.locator('main > header');

  await expect(page).toHaveScreenshot({
    stylePath: [
      './hide-dynamic-elements.css',
      './disable-scroll-animations.css',
    ],
  });

  await expect($hero).toHaveScreenshot({
    stylePath: [
      './hide-dynamic-elements.css',
      './disable-scroll-animations.css',
    ],
  });
});

Auto-Retry Flaky Snapshots

When working with animations or dynamic content, your visual tests can become flaky. Large page snapshots are particularly susceptible.

Playwright can automatically retry failed visual tests for a certain duration, until it finds a valid match. Enable the feature like so:

test('retry snapshots', async ({page}) => {
  await page.goto('https://www.browsercat.com');
  const $hero = page.locator('main > header');

  await expect(page).toHaveScreenshot({
    // retry snapshot until timeout is reached
    timeout: 1000 * 60,
  });

  await expect($hero).toHaveScreenshot({
    // retry snapshot until timeout is reached
    timeout: 1000 * 60,
  });
});

Visual Tests for Generated Images

99.9% of the time, page and element snapshots will cover your use-case. But there are times when you'll want to assert an arbitrary image is consistent across test runs.

For example, perhaps your application generates QR codes or social share cards. Or perhaps you compress and transform user-uploaded avatars. You'll want to ensure this functionality doesn't break.

Use expect().toMatchSnapshot() for this:

import {test, expect} from '@playwright/test';
import {buffer} from 'stream/consumers';

test('arbitrary snapshot', async ({page}) => {
  // generates custom avatars — fun!
  await page.goto('https://getavataaars.com');
  await page.locator('main form button').first().click();

  // download the avatar
  const avatar = await page.waitForEvent('download')
    .then((dl) => dl.createReadStream())
    .then((stream) => buffer(stream));

  expect(avatar).toMatchSnapshot('avatar.png');
});

Compare Snapshots Across Browsers

All of the tests we've written thus far compares the state of your app before and after code changes. But what if you want to compare the state of your app across different browsers and devices?

To accomplish this, we're going to lean on Playwright's "projects" functionality. Projects allow you to define custom test suites with unique configuration. In a mature codebase, you may have quite a lot of these for different devices, environments, and testing strategies.

Let's make some magic!

First, update your playwright.config.ts. If you don't have one yet, create it at the root of your project:

const crossBrowserConfig = {
  testDir: './tests/cross-browser',
  snapshotPathTemplate: '.test/cross/{testFilePath}/{arg}{ext}',
  expect: {
    toHaveScreenshot: {maxDiffPixelRatio: 0.1},
  },
};

export default defineConfig({
  // other config here...

  projects: [
    {
      name: 'cross-chromium',
      use: {...devices['Desktop Chrome']},
      ...crossBrowserConfig,
    },
    {
      name: 'cross-firefox',
      use: {...devices['Desktop Firefox']},
      dependencies: ['cross-chromium'],
      ...crossBrowserConfig,
    },
    {
      name: 'cross-browser',
      use: {...devices['Desktop Safari']},
      dependencies: ['cross-firefox'],
      ...crossBrowserConfig,
    },
  ],
});

Notice how we're specifically configuring the snapshotPathTemplate to store the snapshots for all browsers in the same location. This will ensure that each test compares its snapshots to the same source images.

Next, create a new test file at ./tests/cross-browser/homepage.spec.ts:

import {test, expect} from '@playwright/test';

test('cross-browser snapshots', async ({page, }) => {
  await page.goto('https://www.browsercat.com');
  await page.locator(':has(> a figure)')
    .evaluate(($el) => $el.remove());

  await expect(page).toHaveScreenshot(`home-page.png`, {
    fullPage: true,
  });
});

To avoid a fail, let's initialize our new snapshots:

npx playwright test --project cross-browser -u

And then let's run the tests:

npx playwright test --project cross-browser

Did all of your tests pass? Depending on your environment, they may not have! Different browsers render fonts, colors, and images differently, even when your app is functioning as expected.

If your tests failed, you may need to tweak the maxDiffPixelRatio and threshold options for your snapshots. If you want to debug this issue right now, visit my Ultimate Guide to Visual Testing with Playwright, and learn how to make visual tests more forgiving.

Next Steps...

You're getting pretty good at this! But there's still more to learn. For advice on fine-tuning your snapshot tests and running visual tests in CI/CD, check out my Ultimate Guide to Visual Testing with Playwright.

In the meantime, happy testing!

Getting Started with Snapshot Tests in Playwright

Mike Stop Continues — Mon, 04 Mar 2024 00:00:00 +0000

As your web app matures, it becomes challenging to ensure your GUI doesn't break with any given update. There are a lot of browsers and devices, and countless states for every one of your components. Unit tests ensure your code remains consistent, and E2E tests will ensure your system remains consistent, but neither will catch visual anomalies, layout issues, or platform compatibility issues.

Enter visual testing. Visual tests store snapshots of your app's appearance, and compares them to the current state of your app. No more surprise bugs. No more developing in three different browsers and 12 different breakpoints. Visual testing automates the process, ensuring your app looks exactly right, all the time.

This article covers the basics of visual testing with Playwright. If you're interested in going deeper into visual testing, you'll love my Ultimate Guide to Visual Testing in Playwright.

Let's get started...

1. Install dependencies

If you already have Playwright installed, skip this step.

Otherwise, install your dependencies and create a sample test file:



# Install node dependencies
npm install -D @playwright/test
npm install -D typescript

# Install playwright browsers
npx playwright install

# Create the test file
mkdir -p tests
touch tests/homepage.spec.ts

2. Write your first visual test

Open tests/homepage.spec.ts and add the following code:



import {test, expect} from '@playwright/test';

test('home page visual test', async ({page}) => {
  await page.goto('https://www.browsercat.com');
  await expect(page).toHaveScreenshot();
});

3. Run your visual tests

Run your tests with the following command:



npx playwright test

The first time your run your tests, they will fail with an error message like this:



Error: A snapshot doesn't exist at {TEST_OUTPUT_PATH}, writing actual.

This is expected. Since Playwright has nothing to compare expect(page).toHaveSnapshot() to, it must fail the test. But don't worry: the next time you run your tests, Playwright will compare the current state of the page to the stored snapshot.

So run your tests a second time:



npx playwright test

Success! You should see a message like this:



Running 1 test using 1 worker
  1 passed (6.6s)

If your test failed, take a few minutes to see if you can fix the issue. If not, continue reading. I cover numerous troubleshooting tips later in this article.

Where are snapshots stored?

By default, test snapshots are stored alongside the test file that created them.

When a visual test fails, Playwright will store the "before" image, the "after" image, and a "diff" image in the ./test-results directory. These will come in handy later, for debugging your failed tests.

(Keep reading for advice on configuring the location of these snapshots. You probably don't want test output littering your codebase.)

4. Updating test snapshots

Awesome! Your test now confirms there haven't been any changes to the page, but what if you actually want to make some changes?

To update your snapshots, run your tests with the --update-snapshots (or -u) flag:



npx playwright test -u

This command will run your tests and update the snapshots to match the current state of the page. If you run this now, you should expect a passing result.

A word of caution: This command updates all snapshots. Be very careful, as it's easy to accidentally update snapshots you didn't intend to.

How do I update only some snapshots?

To update the snapshots for a subset of tests, we can filter tests from the CLI. Only matching tests will be updated.



# Update tests with matching file name
npx playwright test -u "**/home*.spec.ts"

# Update tests with matching test name
npx playwright test -u --grep "home" --grep-invert "zzz"

# Updates tests within project
npx playwright test -u --project "chromium"

We'll go deeper into snapshot management later, but for now, let's focus on debugging...

5. Debug visual tests

We know how to update failing tests' snapshots, but what about when a test is failing because of a bug?

To explore this scenario, let's update our test so that our snapshot fails:



test('home page visual test', async ({page}) => {
  await page.goto('https://www.browsercat.com');
  await expect(page).toHaveScreenshot({
    // crop the screenshot to a specific area
    clip: {x: 0, y: 0, width: 500, height: 500},
  });
});

Now let's run the tests, asking playwright to produce an HTML report:



npx playwright test --reporter html

After that command, your browser should have opened to your test report. And it should show a failed result.

Scroll down to the bottom of the page, and you'll see this widget:

The "Diff" view offers a stark comparison between the expected and actual screenshots. However, I find the "Slider" view to be the most useful for actually fixing issues.

How do I interpret the "diff" view?

The "diff" view shows the difference between the expected and actual screenshots.

Yellow pixels vary between snapshots, but they fall within the allowed threshold for differences.

Red pixels fall outside the allowed threshold for differences. By default, even a single red pixel will fail your test.

For advice on configuring these thresholds, visit my Ultimate Guide to Visual Testing with Playwright, and learn how to make visual tests more forgiving.

Do I need to use the HTML report?

No, you don't have to use the HTML report to run visual tests (though I recommend it). Whether you generate an HTML report, failing visual tests will automatically output the "before," "after," and "diff" images to the ./test-results directory.

Often, I'll open the "before" image directly in VSCode while I work. That way I can keep my browser focused on my development app.

6. Use UI mode for quick visual testing

Playwright has an industry-leading "UI Mode" that makes working with tests a breeze.

While it has numerous killer features, my favorite is that it automatically snapshots the test after every single step. This allows you to quickly debug your tests. It also makes it very easy to spot the right places to insert a visual assertion.

To run your tests in UI mode, use the following command:



npx playwright test --ui

I encourage you to explore UI mode, as it has a wealth of useful features.

Next Steps...

You've now got a solid foundation for visual testing with Playwright. But there's so much more to learn!

For advanced tips, configuration guidance, and strategies for managing visual tests in your CI/CD pipeline, check out my Ultimate Guide to Visual Testing in Playwright.

In the meantime, happy testing!

Refactor Playwright Locators Like a Boss

Mike Stop Continues — Fri, 22 Dec 2023 12:00:00 +0000

Your first Playwright automations are likely to be a little... messy. After all, the web is a messy place. But taking automations to production means untangling that ball of spaghetti code, and transforming it into something more clear, performant, and maintainable.

This article explores a few great techniques for refactoring your Playwright locators. And every idea is really quick to implement. After all, there's no point in suggesting optimizations that make your developer life painful!

If you find this post helpful, check out my full Playwright locator deep-dive for even more ideas on optimizing your scripts for speed, scale, and stability.

Let's get started!

Extend Playwright with Custom Selectors

Playwright opens up a world of possibilities with its selector engine, not only by supporting a wide range of built-in selectors but also by empowering users to define their own.

Custom selectors can be tailored to unique application requirements, make scripts more readable, and support complex selection logic that goes beyond the basics.

Here are some ideas:

data-state: Select elements based on a data attribute representing state (e.g., data-state=\"active\") to target specific UI states in a single-page application.
closest: Select the closest ancestor of an element that matches a certain selector—akin to the Element.closest() method in JavaScript.
shadow: Target elements that are within a specific shadow DOM.
rotation: Select an element rotated within a particular range.

To demonstrate the potential of custom selectors in Playwright, let's create the data-state selector described above...

import * as pw from 'playwright';

// register custom selector
await pw.selectors.register('data-state', () => ({
  query(root: Node, selector: string) {
    return root.querySelector(`[data-state="${selector}"]`);
  },

  queryAll(root: Node, selector: string) {
    return Array.from(
      root.querySelectorAll(`[data-state="${selector}"]`)
    );
  },
}));

const browser = await pw.firefox.launch();
const page = await browser.newPage();
await page.goto('https://example.com');

// use the custom selector
const $active = page.locator('data-state=active');
await $active.click();

As you can see, it takes very little to create a custom selector. Since Playwright grants you access to the underlying DOM, the sky's the limit.

Use the Page Object Model

The Page Object Model (POM) is a design pattern widely used in automation for improving maintainability and reducing duplication. A page object encapsulates the behaviors and elements of a specific page in your target site, making it easy to update the code as needed.

Here's an example of what a page object might look like for a login page. Notice how it collects common locators as well as behaviors likely to be repeated in multiple different scripts...

import {Page} from 'playwright';

export class LoginPage {
  page: Page;

  constructor(page: Page) {
    this.page = page;
  }

  get $usernameInput() {
    return this.page.getByLabel('Username');
  }

  get $passwordInput() {
    return this.page.getByLabel('Password');
  }

  get $loginButton() {
    return this.page.getByText('Login');
  }

  public async login(username: string, password: string) {
    await this.$usernameInput.fill(username);
    await this.$passwordInput.fill(password);
    await this.$loginButton.click();
  }
}

Here's how you might use the LoginPage POM:

import * as pw from 'playwright';
import {LoginPage} from './LoginPage';

const browser = await pw.webkit.launch();
const page = await browser.newPage();
await page.goto('https://app.browsercat.com/sign-in');

const loginPage = new LoginPage(page);
await loginPage.login('user@example.com', 'p1a2s3s4word!');
await loginPage.$loginButton.click();

Compose Page Objects from Components

I've found you can extend the wisdom of the Page Object Model further. Consider that most websites are built from reusable components that are repeated on multiple pages. Rather than designing POMs from scratch, instead, you can compose them from components found on page after page.

Here's how we might leverage component objects for AppMenuBar, ChatWidget, and AppFooter:

// Define the separate components
class AppMenuBar {
  constructor(public page: Page) {}
  get $logo () {}
  get $menuButton () {}
}

class ChatWidget {
  constructor(public page: Page) {}
  get $chatToggle () {}
  get $messageInput () {}
  get $sendButton () {}

  async sendMessage(message: string) {}
}

class AppFooter {
  constructor(public page: Page) {}
  get $uptimeStatus () {}
}

// Compose these components into a complete page object
class ContactPage {
  menuBar: AppMenuBar;
  chatWidget: ChatWidget;
  footer: AppFooter;

  constructor(public page: Page) {
    this.menuBar = new AppMenuBar(page);
    this.chatWidget = new ChatWidget(page);
    this.footer = new AppFooter(page);
  }
}

// Use the composed POM
const browser = await pw.chromium.launch();
const page = await browser.newPage();
await page.goto('https://www.browsercat.com/contact');
const contactPage = new ContactPage(page);

// Use the component methods
await contactPage.menuBar.$menuButton.click();
await contactPage.chatWidget.sendMessage('Hello there!');

In the example above, the composed ContactPage object becomes a powerful and convenient abstraction, bundling the interactions for the menu bar, chat widget, and app footer, which are then used in repeated interactions. This approach keeps your scripts DRY (Don't Repeat Yourself) and makes it easy to update component interactions when the UI changes, without having to change each usage.

Next steps

As you can see, these techniques are quick to implement and can have a big impact on the readability, maintainability, and performance of your Playwright scripts. Give one or two a try on an existing project, and see how it goes!

Experimentation is the best way to learn.

If you're looking for more ideas on how to strengthen your Playwright scripts, check out my locator deep-dive next.

Happy automating!

Handle Errors and Fallbacks in Playwright

Mike Stop Continues — Thu, 21 Dec 2023 12:00:00 +0000

All robust automations share one thing in common: they can handle the tides of uncertainty the web throws at them day after day. When developing robust applications on top of Playwright scripts, managing the unexpected is key to winning the day.

This article explores some practical strategies for building resilience into Playwright scripts, ensuring they can handle errors, adapt to varied content, and provide clear control flow.

By the way, if you find this article helpful, check out my complete Playwright locator deep-dive for a detailed look at optimizing your locators for speed, scale, and stability.

Let's get started!

Try Multiple Locator Strategies

When scraping web pages, you may encounter variability in where and how data is presented. For instance, retrieving a star rating from a product page can be challenging because the value could be in a metadata field, within the text of an element, or even graphically represented with icons.

const maybeRatings = await Promise.all([
  page.locator('.rating-value').textContent(),
  page.locator('.star-rating img.star').count(),
  page.locator('meta[name="description"]')
    .getAttribute('content')
    .then((str) => (/\n\.\n+ stars/u).exec(str ?? '')?.[0]),
]);

const bestRating = maybeRatings.reduce((pick, rating) => {
  return pick ?? parseFloat(rating) || null;
}, null);

In this example, the script attempts multiple strategies to capture the rating, testing each result to ensure it looks like an actual rating number.

Quietly Skip Optional Content

Not all content is essential for the successful completion of a script. For example, let's say you wanted to scrape a book product page, but you can't always guarantee the page count is available. In this case, you'd only want the script to fail if critical values (like the book title) weren't found, but to pass otherwise.

const title = await page.locator('.book-title')
  .textContent();
const author = await page.locator('.author-name')
  .textContent();
const pageCount = await page.locator('.page-count')
  .textContent();

const details = {
  title, 
  author,
  pageCount: parseInt(pageCount) || null,
};

assert(!!details.title && !!details.author);

The function demonstrates the selective inclusion of optional data. If present, it enriches the results. But if absent, it doesn't constitute failure.

Clarify Your Control Flow

Control flow can become unclear if scripts are peppered with conditional statements and error handling. Playwright's locators can help us streamline the process.

In the following example, we only interact with the table of contents if it exists.

const $toc = page.locator('aside.contents');

if (await $toc.count() > 0) {
  const $lastLink = await $toc.locator('a').last();
  await $lastLink.click();

  const depth = await page.evaluate(() => window.scrollY);
  console.log(`Scrolled to depth: ${depth}`);
} else {
  console.log('No table of contents found.');
}

This example demonstrates a clear control flow where the presence of a table of contents triggers conditional behavior, but doesn't result in an error otherwise.

Retry Failed Gestures

In some scenarios, you might want a script to retry an action until a certain condition is met, like rolling a digital die until a specific number appears.

const $d20 = page.getByRole('button', {name: 'Roll d20'});
const $result = page.locator('#d20-result');

page.waitForFunction(async () => {
  await $d20.click();
  const roll = await $result.textContent().then((str) => parseInt(str));

  return roll === 20;
});

This code snippet rolls a virtual die, retrying the gesture until the desired outcome is achieved, showcasing a controlled and persistent interaction pattern.

As you can see, these strategies equip your web automation tasks with accuracy and fault tolerance, ensuring smooth interaction with a variety of web pages, content structures, and unexpected circumstances.

Next steps

Now that you've got some new tools in your belt, review your most recent automations and see if you can apply any of these strategies to improve their resilience.

I find the key to making my scripts production-ready is going line-by-line looking for potential failure modes and workarounds. It's a tedious process, but it pays off in the long run.

Remember, check out my complete Playwright locator deep-dive for a detailed look at optimizing your locators for speed, scale, and stability.

Happy automating!

Manage Delays and Async in Playwright

Mike Stop Continues — Wed, 20 Dec 2023 12:00:00 +0000

When you first start out in web automation, it's all about taking the right actions in the right order. But as you advance, timing becomes more and more important.

A great script minimizes the amount of time it spends waiting for changes, and it parallelizes as much behavior as possible. Faster scripts save you time and money, and the work it takes to make them more robust as well.

This article will cover the ins and outs of working with time in Playwright. Cut your scripts from taking minutes to seconds.

And if this post is helpful, visit my deep dive on working with Playwright locators. I cover optimizing your Playwright scripts for speed, scale, and stability in much greater detail.

Let's dive in!

Use the Correct Wait Strategy

By default, Playwright's locators will wait for at least one matching element to appear on the page. In many cases, this is all you need, but the deeper you get into automation, the more frequently this strategy will fall short.

Often, you'll need to watch for other criteria before a particular locator is ready for action. Here's a good example of why you can't always trust the default wait strategy:

// 🟢 GOOD: The default wait works for a single matching element
const $modal = page.locator('#signup-modal');
// 🔴 BAD: But it fails when you need to wait for multiple elements
const $confetti = page.locator('.confetto');

Wait for global state

The "dynamic confetti" problem above can be solved by waiting for the global state to normalize.

// Wait for at least 100 confetti
await page.waitForFunction(async () => {
  return await page.locator('.confetto').count() >= 100;
});
const $confetti = page.locator('.confetto');

page.waitForFunction() will run the function until the return is truthy. This allows you to write very clean code that's equally powerful.

Embrace the State of the Locator

Locators normally resolve when at least one matching DOM node is attached to the DOM. However, Playwright offers a few other convenience methods:

// Attached when added to the DOM (default)
const $attached = page.getByRole('button')
  .waitFor({state: 'attached'});

// Detached when removed from the DOM
const $detached = page.getByRole('button')
  .waitFor({state: 'detached'});

// Visible when attached and has some visible pixels
const $attached = page.getByRole('button')
  .waitFor({state: 'visible'});

// Hidden when detached, `visibility: hidden`, or `display: none`
const $attached = page.getByRole('button')
  .waitFor({state: 'hidden'});

The visible state will frequently come in handy, as a lot of dynamic HTML works by managing visibility rather than dynamically creating new subtrees.

Set a Custom Timeout

By default, Playwright will resolve a locator as soon as it possibly can. But if you're waiting on a dynamically added list of elements, you may prefer to set an extended timeout.

// Wait for a waterfall of injected JS script tags
const $dynamicScripts = page.locator('script')
  .waitFor({timeout: 60_000});

That said, use caution with the timeout approach. If you find yourself leaning on such an imprecise tool, you may want to look twice for better solutions. After all, you can potentially save a lot of time by targeting the precise event that will allow your script to proceed.

Here's one particular strategy...

Wait for DOM Events

Sometimes, it's best to go back to basics and lean on the DOM to ensure the necessary content is loaded before proceeding with your selectors.

For example, let's say that you need a particular image to have loaded before taking a screenshot of the page. You might use the load event to ensure that this is the case:

const $logo = page.getByRole('img')
  .and(page.getByAltText('BrowserCat'));

$logo.evaluate(($el) => {
  return new Promise((resolve) => {
    $el.addEventListener('load', resolve, {once: true});
  });
});

// Logo is loaded!
await $logo.screenshot();

It's worth highlighting the power of Playwright's .evaluate() API. This simple construct gives you direct access to the browser context. Use it to get down to the metal, and control the flow of your scripts directly.

Use Asynchronous Interactions

The longer your scripts become, the more you'll gain by running page interactions simultaneously. Playwright's websocket communication model makes this very performant, and the Locator API is in fact built with asynchronous code in mind.

Let's dive into some strategies worth remembering...

Reuse Locators with Dynamic Content

Playwright's locators are dynamic by design, which means they resolve fresh each time they're accessed. This is particularly handy when dealing with changeable content on a web page.

const $messages = page.locator('#chat li');
let prevMessage = null;

// listen for websocket messages
page.on('websocket', (socket) => {
  socket.on('framereceived', async (event) => {
    const lastMessage = await $messages.last().textContent();
    // verify the new message was added to the chat list
    assert(lastMessage !== prevMessage);
    prevMessage = lastMessage;
  });
});

In this example, we access the locator every time a websocket message is received, then we verify that the new message is added to the chat list.

Notice that the $messages locator is reused without being replaced. This unlocks the possibility of defining your primary locators in the initialization of your script, reusing them as necessary across otherwise complex automations.

Run Tasks in Parallel

Often, scripts wait for one task to complete before starting another. But this isn't a necessary restriction. In Playwright, you can achieve easy parallelism just by employing Promise.all, saving a lot of time in the process.

const result = await Promise.all([
  // Starts a long-running calculation...
  page.locator('#long-running-calculation').click(),
  // Waits for the calculation result to come through...
  page.waitForResponse((res) => /calculation-results/.test(res.url())),
]).then(() => {
  // Retrieve the result
  return page.locator('.calculation-results').textContent();
});

console.log('Calculation completed. Results:', result);

With Promise.all, the script initiates the calculation and waits for the resulting response simultaneously, rather than sequentially. When leveraged strategically, this can shave off precious seconds or even minutes from the total run time.

Leverage a Dependency Graph

Sometimes, you have tasks that depend on others but needn't run in a singular chain. In these cases, you can create a dependency graph by initializing numerous promises, but only awaiting the results at the end of your script. This will ensure that every interaction happens both in the correct order and as swiftly as possible.

This is a somewhat contrived example, but it's not difficult to scale the pattern up. You will assuredly find reasons to do so in your own code.

const doLogin = page.locator('#login').click();
const loadDashboard = doLogin.then(() => page.waitForNavigation({
  url: /dashboard/,
}));
const fetchData = page.waitForResponse((res) => {
  return /data-endpoint/.test(res.url());
});

// Do more stuff here...

const [dashboardLoaded, apiResponse] = await Promise.all([loadDashboard, fetchData]);
console.log('Dashboard has loaded!');
console.log('API data:', await apiResponse.json());

By setting up a structure of chained promises, this example initializes various tasks, but only forces those chains to resolve at the very end. Less time wasted, more money saved.

Next steps

Now that you're brimming with ideas on how to speed up your scripts, go play! Every automation is a little different. The more you experiment, the more you'll learn.

And if you're curious about more ways to optimize your Playwright automations, check out my Playwright locator deep-dive.

Remember, timing is everything. Happy automating!

Simplify and Stabilize Your Playwright Locators

Mike Stop Continues — Tue, 19 Dec 2023 12:00:00 +0000

Who controls the locators, controls the web.

Much of Playwright's power comes from its ability to target and interact with elements on a webpage. But as you well know, the web is a finicky place. Elements come and go, and the slightest change in HTML can break your automation scripts.

In this article, we'll explore strategies for creating simple and stable Playwright locators that can withstand the test of time and changes in web applications. We'll also look at how to optimize your selectors for speed and efficiency.

For a more in-depth look at these concepts, check out my complete deep dive on optimizing Playwright locators.

Let's go!

Optimize Selector Specificity

Refining selector specificity involves finding an ideal equilibrium: you need selectors that are precise enough to unambiguously identify target elements across different page states, yet not so intricate that they shatter with the slightest DOM adjustments.

Steer clear of overly rigid selectors; such fragility means that a trivial HTML modification could render them useless. On the flip side, too loose selectors might lead you down a path of position-dependent logic (like indices) or unwarranted reliance on the web page's current implementation.

We're looking for the Goldilocks zone of search queries.

Here are a few anti-patterns and how to fix them:

// 🔴 BAD: Selectors based on fragile structural assumptions
const $link = page.locator('li:nth-child(3) > a');
// 🟢 GOOD: Selectors grounded in distinguishing features
const $link = page.locator('a[href*="privacy"]');

// 🔴 BAD: Selectors anchored to mutable text
const $button = page.locator('button', {hasText: 'Sign up'});
// 🟢 GOOD: Robust, pattern-based selectors
const $button = page.locator('button', {
  hasText: /(sign|start|subscribe|launch)/i,
});

// 🔴 BAD: Selectors tied to changeable style-specific classes
const $input = page.locator('input.email');
// 🟢 GOOD: Selectors focused on functional attributes
const $input = page.locator('input[type="email"]');

// 🔴 BAD: Excessively specific hierarchical selectors
const $list = page.locator('footer > ul.inline-links > li');
// 🟢 GOOD: Shallow, adaptive nesting
const $list = page.locator('footer:last-of-type li', {
  has: page.locator('a'),
});

To sum up:

Prioritize attributes that capture the essence of the target element, rather than arbitrary qualities.
If your target elements have no stable defining characteristics, anchor your queries to parents or children who do have such qualities.
Keep chain selectors short, and try to only use stable elements within them.
Leverage regular expressions for text matching to accommodate variations.

Prefer Semantic Locator Methods

When it comes to selecting elements, it's always better to prioritize semantic attributes over stylistic or functional attributes. While an element's classes or its location in the DOM are typically the easiest to target, they're also the most prone to change. On the other hand, semantic attributes (such as role, label, or title) change infrequently. And when they do, they tend to change in ways that can be accounted for ahead of time.

Given the benefit of targeting semantic attributes, Playwright provides utility methods for directly accessing them. Use them whenever possible. Here's why:

Semantic methods encourage best practices. As you'll see in the passages that follow, the most durable selector patterns have helper methods.
Semantic methods are typed, improving your IDE experience and alerting you to errors. Plain selector strings can only be debugged at runtime, resulting in more bugs.
Semantic methods are chainable. This dramatically simplifies the queries themselves. It also results in better error messages when queries fail.

Let's run through each method that's currently available:

// Target Test IDs when you have control of the HTML
// E.G. `<section data-testid="delete-modal" />`
const $modal = page.getByTestId('delete-modal');

// Target explicit or implied element roles
// "button" matches `<button>`, `<input type="button">`, or `<div aria-role="button">`
const $button = page.getByRole('button', {hasText: 'Buy'});

// Target text HTML attributes that are unlikely to change
const $input = page.getByLabel('Email');
const $search = page.getByPlaceholder(/^search/i);
const $image = page.getByAltText('Profile Picture');
const $icon = page.getByTitle('Info', {exact: false});

// Or target elements by `innerText`, when you can be sure it's stable
const $dialog = page.getByText(/^confirm/i);

I cannot stress enough how valuable it is to target based on role and data-testid in particular.

When you have control of the DOM, data-testid is a fantastic convention to implement across dev teams. Unlike every other HTML attribute, there is never any reason for data-testid to change, making it the least brittle of all possible selectors.

That said, given that you won't always have control over the HTML of target pages, role is an excellent fallback. As shown in the example code above, role is an inherently forgiving selector. It can continue working even across substantial DOM changes.

Chain Locators, Not Selectors

As described above, it's best to avoid long selector query strings. They're inherently difficult to debug, and they result in less descriptive error messages.

Of course, you're not always going to be able to avoid chaining. Sometimes, you need to target particularly evasive DOM nodes. More often, you simply want to break a page up into subtrees and drill down from an intermediate HTML element. (For example, a list of articles in a blog feed.)

Here are two strategies for designing locators that are both easy to read and easy to debug...

Drill Down Within Subtrees

Chaining locators is like adding layers to a sketch; each additional stroke refines the image. Begin with a broad locator and use methods like .filter(), .first(), .last(), and conditional parameters to progressively narrow down to your target element.

// Find the first article about BrowserCat
const $articles = page.locator('article');
const $aboutBrowserCat = $articles.filter({
  hasText: /BrowserCat/i
});
const $firstArticle = $aboutBrowserCat.first();

This approach separates concerns, ties the locators to logical entities, and keeps the code readable and adaptable. If your layout changes, you might only need to adjust the parent locator instead of unraveling multiple complex strings.

Filter by Content with `{has}` and `{hasNot}`

Sometimes you need to select an element not only by its properties but also by its relation to others. With {has} and {hasNot} parameters, you can define these relationships clearly, creating a robust context for your selectors.

// Find the first article with an image
const $articles = page.locator('article');
const $withImages = $articles.filter({ 
  has: page.locator('img'),
});
const $firstArticle = $withImages.first();

These parameters act as assertions about the presence of certain elements within a parent. This increases the number of stable, unique attributes you can leverage in creating good query patterns.

Next steps

We've touched on several key practices for creating simple and stable Playwright locators that can withstand the test of time and changes in web applications. I go much deeper on performance, error handling, and refactoring your selectors in my Playwright locators deep-dive. Check it out.

Until next time, happy automating!

What are Playwright Locators and Selectors?

Mike Stop Continues — Mon, 18 Dec 2023 12:00:00 +0000

Playwright is a Swiss Army knife for browser automation, and nearly all of Playwright's expansive functionality is based on two core tools: selectors and locators. They're essential for anyone serious about working with Playwright, and they're the focus of this article.

In the following post, we'll cover what they are, why they matter, and how to use them effectively. Once you understand the basics, visit my deep dive on strengthening your Playwright locators. It's a great resource for truly mastering these tools.

Let's get started!

What are Playwright Selectors?

Playwright selectors are search queries used to match elements within the page's DOM. They leverage various strategies such as CSS, XPath, textual content, pseudo-selectors, and even custom properties you define. Selectors are strings, and they are the foundation of Playwright's locators, though they don't enable interaction by themselves.

Here are some examples:

// create a locator using css
const $buttonByCSS = page.locator('css=button');

// create a locator using xpath
const $buttonByXPath = page.locator('xpath=//button');

// create a locator using psuedo-selectors
const $buttonByRole = page.getByRole('button');

Playwright also allows the registration of custom selectors, leveraging any logic you can dream up. At first blush, this might seem like overkill, but as code matures and you develop domain logic, custom selectors can dramatically simplify your code.

Here's a brief taste:

// target elements currently within a particular shadow DOM
await selectors.register('shadow', createShadowSelector);

// query all options within a custom select web component
const $options = page.locator('shadow=x-custom-select option');
await $options.count();

We'll return to this technique later with a complete example.

What are Playwright Locators?

Locators are a powerful abstraction built on top of selectors. They provide a clean interface for selecting elements, interacting with them, and traversing the DOM subtree within them. You use them constantly in Playwright.

As an improvement over the now-deprecated $() API, they also come with some unique advantages. First, they are always up-to-date. A selector that targets a list of elements will always select the full list, no matter how many items have been added, removed, or changed.

And second, locators will wait until the selector pattern finds a match on the page. This is incredibly useful when working with dynamic content, as you no longer have to build arbitrary or complex timeout logic into your scripts.

We'll cover locators in great detail later, but here's a quick sample:

// find the button with `name="submit"`
const $submitButton = page.getByRole('button', {name: 'submit'});
// or...
const $submitButton = page.locator('role=button', {name: 'submit'});

// click it!
await $submitButton.click();

Locators can also be chained to increase specificity without having to resort to long selector queries. Example:

const $taskList = page.locator('ul#tasks');
const $taskLinks = $taskList.locator('a');
const $completedTasks = $taskLinks.filter({
  has: page.locator('input[type="checkbox"]:checked'),
});

Next steps

By now, you should have a solid understanding of Playwright locators and selectors. To dive deeper, make sure you check out my everything guide to strengthening your Playwright locators.

Happy automating!

Block Ads in Playwright with an Adblocker

Mike Stop Continues — Fri, 01 Dec 2023 00:00:00 +0000

Playwright is such a powerful technology because it gives you programmatic access to a full web browser. However, that means it also suffers from the same weaknesses as the browser. Anything that slows down a human browsing the web also slows down your scripts.

Not least among these slow-downs are online ads, trackers, and other annoyances. By preventing these third party resources from loading, we can improve performance, improve stability, and decrease cost.

This article covers how you can efficiently reduce the load of ads and trackers within Playwright scripts for an optimal automation experience. If you're interested in going deeper into resource management and optimization, you'll love my Playwright resource deep-dive.

Let's get started...

Block Ads With Built-In Methods

A straightforward method to stop ads and trackers is to block them by domain. For websites that have minimal dependencies on third-party domains, this is an effective and no-extra-dependency solution:

// watch all network activity
await page.route('**/*', (route) => {
  // abort requests from known ad domains
  if (route.request().url().includes('cdn.ads.com')) {
    route.abort();
  } else {
    route.continue();
  }
});

This is an excellent first step in managing ads during a Playwright job, but it's also quite limited. Firstly, it requires that you keep track of all the third-parties you want to block on your target site. And second, it requires that you know ahead of time which websites your script will run on. After all, your tool might load sites selected by your end users, or it might crawl multiple arbitrary sites during runtime.

Embed an Ad Blocker

Just like in a real browser, you can run an adblocker in your Playwright scripts, leveraging huge open-source blacklists of known offenders.

While this strategy depends on an external library, it's by far the best way to block wasteful web traffic in your automations.

From all the options out there, I recommend @cliqz/adblocker-playwright. It's a great library with clean docs and great config options (if you even need them). Let's set it up...

Getting Started with `@cliqz/adblocker-playwright`

It all starts with installing the package:

npm install --save @cliqz/adblocker-playwright

Then set it up like so:

import * as pw from 'playwright';
import { PlaywrightBlocker } from '@cliqz/adblocker-playwright';

const browser = await pw.chromium.launch();
const context = await browser.newContext();

const blocker = await PlaywrightBlocker.fromPrebuiltAdsAndTracking(fetch);

blocker.enableBlockingInContext(context);

By tapping into an up-to-date list of ad sources and trackers, this strategy efficiently filters out unwanted noise. And if you need more control, well, you got it...

Pick Your Favorite Blacklists

Perhaps you've got a custom blacklist. Or perhaps you want to include a few of the open-source lists maintained by good webizens. @cliqz/adblocker supports leveraging both local and remote blacklists.

Here's an example:

import { readFileSync } from 'fs';
import { PlaywrightBlocker } from '@cliqz/adblocker-playwright';

const blocker = await PlaywrightBlocker.fromLists(fetch, [
  'https://easylist.to/easylist/easylist.txt',
]);

blocker.enableBlockingInContext(context);

Interested in picking some custom lists? I suggest starting with EasyList, then if you discover gaps, search the web. There are many, many options.

Next Steps...

These ad-blocking strategies with Playwright pave the way for cleaner automation workflows, better performance, and enhanced security. But this is only one tool in your Playwright optimization toolkit. Be sure to explore my in-depth guide on managing Playwright resources. It has a lot of great tips packed as densely together as I could fit them.

In the meantime, happy automating!

Block Inline Styles and Scripts in Playwright Scripts

Mike Stop Continues — Fri, 24 Nov 2023 00:00:00 +0000

Speed is a key consideration in web automation and testing. While network requests often play a key role in performance, inline resources can have an equally detrimental impact. And they're often overlooked. After all, inline stylesheets and scripts can slow down your Playwright scripts as easily as external ones.

In this article, we'll explore practical techniques to block inline styles and scripts that don’t serve your automation’s purpose, thus speeding up your Playwright scripts significantly.

For a comprehensive guide to managing resources that goes well beyond inline resources, be sure to visit my Playwright resource guide.

Block Inline Scripts Before Load

Playwright’s .addInitScript() method is the gatekeeper that lets you run custom JavaScript code before anything else when a navigation event occurs. You also have the flexibility to lace your code with this interception on a Page or Context level:

// destroy all scripts on this page before they load
await page.addInitScript(() => {
  document.addEventListener('DOMContentLoaded', () => {
    document.querySelectorAll('script')
      .forEach((el) => el.remove());
  });
});

Block Inline Styles Before Load

Often overlooked, inline styles can be just as cumbersome, especially when dealing with large DOM trees or complex CSS animations. Blocking them ensures the browser won’t spend unnecessary cycles parsing and applying styles irrelevant to your script's goals:

// destroy all stylesheets on _all_ pages before they load
await context.addInitScript(() => {
  document.addEventListener('DOMContentLoaded', () => {
    document.querySelectorAll('style')
      .forEach((el) => el.remove());
  });
});

Block Other Inline Resources Before Load

This strategy isn't limited to obvious bottlenecks like CSS and JS. You can use the same method to gut the DOM of huge wastes of resources. This is especially useful if you're only focused on scraping data, not screenshots or other visual actions.

// remove huge swathes of the DOM before they load
await page.addInitScript(() => {
  document.addEventListener('DOMContentLoaded', () => {
    document.querySelectorAll('.sidebar, .footer')
      .forEach((el) => el.remove());
  });
});

While the idea of aggressively pruning the DOM for speed gains is appealing, it comes with caveats. Broadly eliminating parts of a page can lead to unexpected results, especially if you're removing elements that may serve as dependencies for others. On some websites, it may also trigger anti-bot measures. As always, test your work!

Block Dynamically-Loaded Resources

While the DOMContentLoaded event tackles resources available at load time, dynamic pages often inject scripts and styles after the initial load. You can get around this by leveraging the MutationObserver API, which allows you to intercept and manage assets as they appear:

await page.addInitScript(() => {
  const observer = new MutationObserver((mutations) => {
    for (const mutation of mutations) {
      mutation.addedNodes.forEach((node) => {
        if (['SCRIPT', 'STYLE'].includes(node.tagName)) {
          node.remove();
        }
      });
    }
  });

  observer.observe(document.documentElement, {
    childList: true,
    subtree: true,
  });
});

Next Steps...

The approaches outlined here are but one facet of Playwright’s robust capability set. Harnessing these strategies propels you towards leaner, faster-executing scripts, steering clear of unnecessary rendering or computations.

Nonetheless, the journey to master web automation doesn’t stop with inlining. Blocking network traffic, embedding an adblock, and tweaking the browser runtime can all have a huge positive impact on your scripts' performance. For more info, read my Playwright resource guide. It's very thorough!

Speed Up the Headless Browser in Playwright

Mike Stop Continues — Fri, 17 Nov 2023 00:00:00 +0000

For developers immersed in web automation, Playwright is a vital ally... but it can be a pretty heavy process to run. As you begin to rely more on headless browsers, it becomes important to be able to make those scripts faster and more stable.

Thankfully, there are a lot of powerful levers to pull for performance in Playwright, depending on your situation. This article dives into some of the most potent levers--configuring the browser through Playwright config, CLI args, and feature flags.

If you find these tips powerful, but perhaps not right for your use-case, be sure to check out my top tips for removing network bottlenecks and heavy resource loading in this resource management deep-dive.

But without further ado, let's go through the big levers you can pull for performance in Playwright...

Disable All Javascript

JavaScript is the cornerstone of modern web interactivity, but it can become a liability in headless browser sessions that simply don't require it. By disabling JavaScript, you can significantly accelerate your scripts, skipping over:

Hydrating the React/Svelte/Vue app
Launching pop-ups
Loading third-party scripts
Replacing placeholder images with full-size images
Initializing auto-play videos
Triggering a waterfall of subsequent JavaScript requests
Destroying content by enabling a paywall
and on and on...

Here's how to switch it off for all pages loaded within a Playwright context:

const browser = pw.chromium.launch();
const context = browser.newContext({javaScriptEnabled: false});

If your use-case doesn't require JS to load on your target pages, this is a powerful technique. But be warned! Disabling JavaScript might trigger a website's anti-bot defense system. After all, many less-sophisticated web crawlers don't run JavaScript either.

Tweak Chromium CLI Args

By default, Playwright tries to ensure its headless browsers behave maximally similar to a user-driven browser.

But what if your use-case doesn't leverage every aspect of the browser's capabilities? If not, it may be worth exploring the CLI args your chosen browser makes available.

This is especially valuable for Chromium-based browsers, which provide dozens of options, including offering CLI access to feature flags (chrome://flags). This gives you tremendous control over the browser process.

Here are some of the best flags to draw from...

Disable Sandboxing

There's one surefire way to improve the performance of Chromium-based browsers no matter what your use-case is. That's to disable security sandboxing. But be warned--doing so may expose your execution context (Docker container, Lambda function, etc.) to malicious code.

Even if you aren't including user-generated data in your scripts, you may still be vulnerable to attack. After all, third-party dependencies may carry code designed to sniff for sensitive data and access. While this possibility is unlikely, it is important to be aware of the consequences.

Nevertheless, it's a powerful tool under the right circumstances.

const browser = pw.chromium.launch({
  args: [
    // Use with caution!
    '--no-sandbox',
    '--disable-setuid-sandbox',
  ],
});

Disable Visual Rendering Features

If you don't care about taking screenshots, recording videos, or generating PDFs, you can probably save big by disabling Chromium's visual components.

This trick is just as good as easy as the previous one, largely boiling down to a single switch:

const browser = pw.chromium.launch({
  args: [
    // Huge gains when you don't need visual output
    '--disable-gl-drawing-for-tests',
  ],
});

Pick the Right Flags for You

There are way more CLI args than I can go through here, especially because which flags you set depends on your use-case.

For example, there are flags for antialiasing, for memory allocation, for browser features (like extensions, alerts, etc.), and more. Which you switch depends on what you're working on. And truth be told, most won't be as powerful as the few I've already listed here.

At any rate, experiment!

Next Steps...

With these strategies, you're positioned to extract the most out of Playwright and your headless browser environment. Saving even a few seconds per run amounts to hours, days, and years of cost-savings on server costs. And even better, it improves the stability of the features your users depend on.

All that said, Playwright offers many more options for optimization, such as blocking network traffic and preventing inline assets from loading. If you're eager to delve even deeper, see my comprehensive guide.

Happy automating!

Forem: Mike Stop Continues

How to Run Snapshot Tests in CI/CD with Playwright

1. Configure Playwright for CI/CD

2. Run your tests in CI/CD

3. Store snapshots between runs

4. Trigger a snapshot update

5. Speed up your CI/CD pipeline... a lot!

Next Steps...

Configuring Snapshot Tests in Playwright

Custom Snapshot File Names

Custom Snapshot Directories

Make Visual Tests More Forgiving

Configure forgiveness globally

Override forgiveness per-assertion

How do I tune these options?

Next Steps...

Advanced Snapshot Testing in Playwright

Page vs. Element Snapshots

When should I use page snapshots?

When should I use element snapshots?

Working with Page Snapshots

Cropping Page Snapshots

Snapshot the Entire Page

Scroll Before Taking a Page Snapshot

Working with Element Snapshots

Test Element Interactivity

Test Element Responsiveness

Advanced Snapshot Techniques

Masking Portions of a Snapshot

Keeping Styles Constant During Snapshots

Auto-Retry Flaky Snapshots

Visual Tests for Generated Images

Compare Snapshots Across Browsers

Next Steps...

Getting Started with Snapshot Tests in Playwright

1. Install dependencies

2. Write your first visual test

3. Run your visual tests

Where are snapshots stored?

4. Updating test snapshots

How do I update only some snapshots?

5. Debug visual tests

How do I interpret the "diff" view?

Do I need to use the HTML report?

6. Use UI mode for quick visual testing

Next Steps...

Refactor Playwright Locators Like a Boss

Extend Playwright with Custom Selectors

Use the Page Object Model

Compose Page Objects from Components

Next steps

Handle Errors and Fallbacks in Playwright

Try Multiple Locator Strategies

Quietly Skip Optional Content

Clarify Your Control Flow

Retry Failed Gestures

Next steps

Manage Delays and Async in Playwright

Use the Correct Wait Strategy

Wait for global state

Embrace the State of the Locator

Set a Custom Timeout

Wait for DOM Events

Use Asynchronous Interactions

Reuse Locators with Dynamic Content

Run Tasks in Parallel

Leverage a Dependency Graph

Next steps

Simplify and Stabilize Your Playwright Locators

Optimize Selector Specificity

Prefer Semantic Locator Methods

Chain Locators, Not Selectors

Drill Down Within Subtrees

Filter by Content with {has} and {hasNot}

Next steps

What are Playwright Locators and Selectors?

What are Playwright Selectors?

What are Playwright Locators?

Next steps

Block Ads in Playwright with an Adblocker

Filter by Content with `{has}` and `{hasNot}`

Getting Started with `@cliqz/adblocker-playwright`