Forem: Chromatic

Introducing Chromatic’s Cypress integration

Chromatic — Thu, 07 Dec 2023 18:12:05 +0000

TL;DR: We’re building an integration that brings Chromatic into Cypress. That gives you all of Chromatic’s visual testing capabilities, scanning every possible state of UI, within your E2E tests. All it takes to get started is a couple lines of code. Join early access now.

Every team needs the confidence that their apps work and look great for their users. Today, we’re sharing how you gain get that through Chromatic’s new integration for Cypress: E2E Visual Tests. This lets you bring Chromatic’s visual testing into your existing E2E workflow, and all it takes to get started is two tiny changes in your code.

Read on for an overview of how E2E Visual Tests for Cypress works, and how you can try it now in early access.

Using Playwright? You’ll want to head here 👉

Why integrate Chromatic and Cypress?

As a visual testing tool, Chromatic doesn’t just capture an image of each page you test. Instead, E2E Visual Test snapshots are living re-creations of a page’s DOM, styling, and assets. Whenever you’re reviewing test results, you receive high-fidelity renderings that are fully inspectable. This makes it painless to reproduce and confirm changes, because you’re seeing exactly the same UI as your users.

Snapshots are taken in a controlled cloud environment, using consistently-versioned, modern browsers. Whether you’re testing in one browser or four, they run in parallel so your test suite stays fast.

Every snapshot taken is saved to Chromatic’s CDN. Having your snapshots on Chromatic’s servers, rather than in your local filesystem, means that you’re not tracking and pushing up to your repo what can grow to be an enormous set of images!

Finally, by integrating Chromatic into your E2E tests, you gain all the benefits of the Chromatic platform. This includes visual testing down to the pixel, a single location for designers and other stakeholders to sign-off on work, plus a published library of all your snapshots. All of this happens out of the box just by integrating Chromatic into Cypress—without any other changes to your tests.

Get started with two small steps

Firstly: in cypress/support/e2e.js, import Chromatic’s test-archiver for Cypress:

// cypress/support/e2e.js
import '@chromaui/test-archiver/cypress/support';

Then, in cypress.config.js, add our archiveCypress task:

// cypress.config.js

const { defineConfig } = require("cypress");
const { archiveCypress } = require('@chromaui/test-archiver/cypress');

module.exports = defineConfig({  
  e2e: {
    setupNodeEvents(on, config) {
      // implement node event listeners here
      on('task', {        
        archiveCypress
      })
    },
  },
});

That’s it! You’re now set up to start benefiting from Chromatic in all of your Cypress tests, without adding any more code! Chromatic will now take a snapshot at the end of every Cypress E2E test.

Here’s how it works

Imagine you’re a developer on an online shopping product. You’re tasked with verifying how data flows from the backend to the frontend for a critical product detail page.

To begin, let’s just make sure we can successfully visit the page. We’ll use Cypress’s built-in cy.visit() command to navigate to the product page. Then, we’ll use cy.title().should() to verify that the page title renders as expected:

describe('My First Test', () => {
  it('Visits product page', () => {
        // 👇 go to the Amazon product page
    cy.visit('https://www.amazon.com/Tungsten-Cube-Biggest-Size/dp/B07WK9WLZ8/')
        // 👇 verify the page title
        cy.title().should('eq', 'Amazon.com: The 4" Tungsten Cube - Biggest Size : Toys & Games')
  })
})

There’s not much to this test. So, presumably, the page will render and the test will pass. However, in addition to the test being green, you now also gain a visual snapshot of the page taken at the completion of the test. You can view that snapshot in Chromatic’s UI:

Remember, though, that you were tasked with verifying how data flows from the backend to the frontend, not just that the page loads. To do that, we could add to the initial test and write new assertions to ensure every value on the page—price, description, product title, etc.—match our expectations. But that would quickly grow verbose.

Instead, you can verify all those details in a single glance with the snapshot you just created. And if the data ever changes, Chromatic will catch it and highlight it in a visual diff for you to verify or deny. Now you’re testing that the data flows correctly and that the appearance of the page is correct, too. And with much less code!

Now, another example. This time, we want to test the interaction of the search bar on the product page.

To begin, you start again by visiting the product page URL in Cypress. But once there, you find the main search field on the page and type in a search:

describe('My First Test', () => {
  it('Searches from product page 1', () => {
        // 👇 go to the product page
    cy.visit('https://www.amazon.com/Tungsten-Cube-Biggest-Size/dp/B07WK9WLZ8/')
        // 👇 verify the page title
        cy.title().should('eq', 'Amazon.com: The 4" Tungsten Cube - Biggest Size : Toys & Games')
        // 👇 find the main search field and type in it
    cy.get('.nav-search-field > input').type('rubik’s cube')
  })
})

Ok, cool. We should get some passing tests again and a snapshot of the final state of the UI, showing ‘rubik’s cube’ typed into the search field.

But what if we want some snapshots taken during that process? Easy! You can use cy.takeChromaticArchive() to take a manual snapshot at specific points during your tests. So let’s use this method to take a snapshot when we first focus the search field to see what that looks like:

describe('My First Test', () => {
  it('Searches from product page 2', () => {
        // 👇 go to the product page
    cy.visit('https://www.amazon.com/Tungsten-Cube-Biggest-Size/dp/B07WK9WLZ8/')
        // 👇 verify the page title
        cy.title().should('eq', 'Amazon.com: The 4" Tungsten Cube - Biggest Size : Toys & Games')
        // 👇 find & focus the search field
    cy.get('.nav-search-field > input').focus()
        // 👇 tell Chromatic to take a snapshot here
    cy.takeChromaticArchive()
        // 👇 finish the test as before
    cy.get('.nav-search-field > input').type('rubik’s cube')
  })
})

When this test runs, Chromatic will capture two snapshots: the one we’ve manually taken of the focused search field and the one automatically taken at the end of the test.

One line of code to snapshot two different states of the page, plus the functional tests you already had? Not bad.

Sign up for early access

Help us bring Chromatic’s Cypress integration over the finish line by signing up for early access! Try it out and give feedback so that we can ship the best package possible.

👉 Sign up here 👈

How to run visual tests in 2023

Chromatic — Fri, 25 Aug 2023 12:47:37 +0000

One of the key inspirations for Storybook was a Chromatic post from 2017. Below is an updated version of the article, current to the frontend landscape today.

Testing is crucial for developing high-quality software. But when testing how UI looks and renders, it’s not really possible to write tests to verify 'correctness'.

The issue stems from the fact that the most important details of your components are hard to express programmatically. Your choice of HTML tags/classes, or the text in the output, for example, don’t provide the full picture that's required for a test to assert whether UI will look good (or not) to human users.

Since the first version of this article emerged in 2017, the discipline of visual testing has taken over the frontend world. It's been embraced by thousands upon thousands of teams to ensure that how their app looks forms a core consideration within its overarching functionality.

In this updated article, we reflect on the journey of UI testing over the past several years. We cover the pros and cons of different historic techniques that have been tried for UI tests, and explore the major impact of the emergence of Storybook, enabling devs to focus solely upon testing the components/states that require their attention the most.

Unit testing UI

A unit test isolates a module and verifies its behaviour by applying a set of inputs. It does this by comparing the module’s output to an expected result.

Unit testing is desirable because testing a module in isolation – rather than the whole system – streamlines the process of covering tricky edge cases. It lets you pinpoint the locations of issues when they occur.

However (and crucially): unit tests don’t have eyeballs.

While we can specify the inputs of our unit tests (like React props and context ), it’s harder to specify the output in a way that’s robust to the minor changes within component implementation. For example: changing from flexbox to grid will cause many tests to fail, even when the result looks exactly same.

There are great tools like Testing Library that render a single component in a single state and then programmatically inspect the resulting DOM. However, the salient details of the rendered HTML – like color, layout, and scale – are hard to express. So, the tests end up brittle (over-specified) or insufficient (under-specified).

The core issue, though, is that most components’ inherent complexity is visual : the specifics of how generated HTML and CSS appears on screen. And it has never been easy to write a test for appearance.

Snapshot testing

One solution to this problem came in the form of snapshot tests.

Snapshot tests highlight changes in UI by comparing HTML outputs with a recorded baseline. You can then verify whether these changes are intentional.

This approach acknowledges that UI outputs are difficult to specify in unit tests, so it informs developers when any code pertaining to the UI is changed.

Unfortunately, snapshot testing suffers from the same brittleness as unit tests. Minor details like changing a class name on a div can trigger the test to fail, meaning your tests require constant maintenance and checking in order to root out false positives.

...and then came Storybook

Storybook provided a visual approach to traditional testing techniques. It lets you view any component in any specific state, and then capture it as a test case by writing a story. With > 20m downloads each month, it continues to serve that function today!

The key idea behind Storybook is a that a developer can manually verify a story by rendering it within a browser. In practice, this approach is much easier than sifting through the false positives of unit/snapshot tests, updating test cases to match minor UI changes, or working overtime to make tests pass again.

Stories = test cases

Visual tests share the same setup, execution and teardown steps as any other type of testing. However, the final verification is left to the developer.

In Storybook, writing a test case is the same as writing a story:

// src/components/TaskList.stories.jsx

export default {
  component: TaskList,
};
export const Default = {
  args: { 
    tasks: inboxTasks,
  },
};

To manually verify the TaskList in the "Default" state, you would browse to the relevant story to render it in isolation. You can then see that it looks OK and then you’re done!

Automated checks

Storybook's visual testing requires you to capture the relevant component states as stories and verify each one.

But: what happens when you have thousands of stories? Checking each one would take forever.

That's why we created Chromatic, answering precisely that problem by handing developers an automated visual testing tool that runs in CI. Over the past 5 years, it’s been battle-tested by organizations like Adobe, the BBC, Monday.com, Peloton, and Mercado Libre.

The core principle of Chromatic is that you only need to focus on visible changes to the UI. You’re still involved in confirming whether changes are intentional, but a machine has already done the first pass at finding these changes for you.

It works by automatically taking screenshots of every story to compare them – pixel-by-pixel – to past versions of the UI. Next, it builds a changeset of stories that have visual differences and shares them with you for review.

🎉 Storybook’s next big thing

The Storybook team are currently working on a new addon that will let you bring automated visual testing directly into your local Storybook instance: the Visual Tests addon.

It pinpoints visual changes in stories across browsers and viewports and is powered by the same technology as Chromatic.

The Visual Test addon is currently in development with six active maintainers. Your feedback can help bring it to life faster. Sign up below to get early access (with free usage) and project updates:

👉 Sign up now

How to visually test UI using Playwright

Chromatic — Wed, 23 Aug 2023 17:51:21 +0000

Playwright is an excellent tool for verifying that your app’s most important user flows work as expected. But did you know that Playwright can also help you verify that those flows look as expected too? This process is called visual regression testing, or visual testing for short. And it’s super helpful for verifying all kinds of UI details: responsive design shifts, cross-browser/device differences, localization, dynamic content, and more.

In this post, we’re going to walk through how to run visual regression tests using Playwright. We’ll add visual testing to an example E2E test and walk through the typical workflow when changes occur. For this exercise, we’ll assume that you have Playwright configured and running at least one E2E test already.

How does visual testing work?

Playwright’s visual testing revolves around a process of screenshot comparison. When the test begins, Playwright takes a screenshot of whatever is being tested and stores this as the baseline. Then, whenever you change your code, Playwright takes another screenshot and compares it against the baseline.

If the changes look correct, you can accept them and then set the new baseline. Otherwise, you can continue updating your code and accept it when you’re happy with the results.

Create screenshots

To begin visual testing in Playwright, let’s start with an example E2E test which opens a dialog and then clicks the close button within it:

// tests/dashboard.spec.ts
import { test, expect } from "@playwright/test";

test("Dashboard", async ({ page }) => {
  await page.goto("/dashboard/acme");

  await expect(page).toHaveTitle(/Acme Dashboard/);

  const expandButton = await page.locator(
    ".main .card:nth-child(0) .btn-expand"
  );

  await expandButton.click();

  const dialog = await page.locator(".dialog");

  const closeButton = await dialog.locator(".btn-close");

  await closeButton.click();
});

Playwright provides the page.screenshot API to take screenshots of the page you’re testing. To use it, add a line that calls the function at the part of the flow where a screenshot should be taken:

// tests/dashboard.spec.ts
import { test, expect } from "@playwright/test";

test("Dashboard", async ({ page }) => {
  await page.goto("/dashboard/acme");

  await expect(page).toHaveTitle(/Acme Dashboard/);

  const expandButton = await page.locator(
    ".main .card:nth-child(0) .btn-expand"
  );

  await expandButton.click();

  const dialog = await page.locator(".dialog");

  // 👇 Take a screenshot once the dialog is located
  page.screenshot({ path: 'latencyExpanded.png' });

  const closeButton = await dialog.locator(".btn-close");

  await closeButton.click();
});

page.screenshot has a couple of options that may be useful to you. You can capture the full page height (instead of just the viewport height) and you can configure where the screenshot is saved:

await page.screenshot({
  fullPage: true, // Capture full page height
  path: 'screenshot.png', // Provide save location
});

You can also screenshot a particular element instead of the entire page:

await page.locator('.dialog').screenshot(...);

Once you’ve updated your test, run your test command to save a screenshot:

yarn playwright test

In our example above, the screenshot will save in the root of the project:

Open the screenshot to confirm that it looks correct:

Now, commit both your code change and the newly-created screenshot. Then push your commit, making the screenshot your baseline.

Run visual tests

Next, we’ll use that baseline screenshot to verify a change.

First, create a new branch and make a code change. For this example, we’ll change our primary chart color:

- fontFamily: 'system-ui, san-serif',
+ fontFamily: 'Inter, system-ui, san-serif',

After making the change, run your test command again to save a new screenshot. Once more, open the screenshot to confirm it looks correct.

Commit your changes and make a PR for your new branch.

Now, when we review that PR, we can compare the previous baseline screenshot and the modified one:

Automate tests in CI

In the workflow so far, we’ve created the screenshots on a local machine and then committed them to your repo. This can cause problems if you're working with other developers on the same E2E tests, as device differences like installed fonts could cause unnecessary image differences.

To mitigate this, take your screenshots within your continuous integration (CI) environment, and let your CI job handle the screenshot and committing process for you. Then, you can review the results as part of the PR review, as before.

Here’s an example using GitHub Actions:

# .github/workflows/e2e.yml
name: E2E tests
on: push
permissions:
  contents: write
jobs:
  E2E:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v3
        with:
          node-version: '16'
      - uses: actions/checkout@v3
      - name: Install dependencies
        run: yarn
      - name: Install Playwright Browsers
        run: yarn playwright install --with-deps
      - name: Run tests
        run: yarn playwright test

      - name: Update E2E screenshots
        run: |
          git config --global user.name 'Your Name'
          git config --global user.email 'you@example.com'
          git commit -am "Update screenshots"
          git push

The screenshots are still part of your repo (which can take up a lot of space if you have many tests), but now they’re created in a shared, consistent environment, eliminating any device differences.

Debugging an issue can be difficult in this flow. A static screenshot often doesn’t provide enough information, and debugging the real page requires navigating through a staging environment and recreating any interactions that took place.
Reviewing and collaborating with non-developers can also be tricky, because it all happens within the PR experience, which many are unfamiliar with.

Debuggable snapshots and collaborative review

You can take this workflow even further when you integrate Playwright with Chromatic, a cloud-based visual testing platform.

Here’s how to use it in our example above:

// tests/dashboard.spec.ts
// ➖ Remove this line
// import { test, expect } from '@playwright/test';
// ➕ Add this line
import { test, expect, takeArchive } from "@chromaui/test-archiver";

test("Dashboard", async ({ page }) => {
 await page.goto("/dashboard/acme");

 await expect(page).toHaveTitle(/Acme Dashboard/);

 const expandButton = await page.locator(
   ".main .card:nth-child(0) .btn-expand"
 );

 await expandButton.click();

 const dialog = await page.locator(".dialog");

 // 👇 Take a screenshot once the dialog is located
 // ➖ Remove this line
 // page.screenshot();
 // ➕ Add this line
 await takeArchive(page, testInfo);

 const closeButton = await dialog.locator(".btn-close");

 await closeButton.click();
});

And then the CI job (using GitHub Actions here) would be:

# .github/workflows/e2e.yml
name: E2E tests
on: push
jobs:
  E2E:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v2
        with:
          fetch-depth: 0
      - name: Install dependencies
        run: yarn
      - name: Install Playwright Browsers
        run: yarn playwright install --with-deps

      # 👇 Run your E2E tests *before* running Chromatic for your E2E test archives
      - name: Run tests
        run: yarn playwright test

      # 👇 Run Chromatic for your E2E test archives
      - name: Publish E2E Archives to Chromatic
        uses: chromaui/action@v1
       with:
         projectToken: ${{ secrets.CHROMATIC_ARCHIVE_PROJECT_TOKEN }}
         buildScriptName: build-archive-storybook

Now, instead of creating a static screenshot, running your test will create a fully debuggable snapshot of the page that you can open in the browser. Those snapshots are created in a dedicated, specialized environment—making the process fast and consistent—and saved in Chromatic’s database instead of cluttering your project’s repo. Once in Chromatic, snapshots are easily shared with all of your project's stakeholders and contributors. And there’s a web app purpose-built for reviewing visual tests, which integrates with your git provider.

Right now, Chromatic's Playwright E2E testing integration is available to try in early access, with free usage during the beta period.

Sign up now for early access ≫

Introducing automated visual testing to Playwright

Chromatic — Wed, 16 Aug 2023 16:11:38 +0000

TL;DR: Playwright's snapshot tests are a powerful option for visual testing. Soon, you'll be able to take them even further by integrating Chromatic into Playwright, letting you run automated visual testing within your existing E2E tests. Sign up now for early access.

Playwright is an essential tool for testing user flows to ensure they actually work (e.g. "sign up" or "add to cart" flows).

However, these E2E tests only verify the logic. You still need to manually check each page to confirm that it "looks correct." In other words, you also need be on the lookout for visual bugs.

Testing for visual bugs is tedious. UIs are constantly changing, making it challenging to keep up with broken layouts, missing assets, and cross-browser rendering differences. Additionally, modern UIs are complex, with hundreds of components and thousands of states to consider. Writing assertions for every single visual detail is simply impossible.

Imagine if you could ensure that user flows work correctly and look right, without needing to write more tests. We're building a tool that automatically runs visual tests on every step of your E2E test pages, while Playwright drives the browser.

Read on to learn more or sign up now for early access.

What is visual testing?

If E2E tests verify that things work right, visual tests verify that things look right.

Visual tests detect bugs in UI appearance by capturing pixel-perfect snapshots of the rendered UI, including styling and static assets. These snapshots are then compared to baseline snapshots as you commit code, helping to catch UI bugs.

These are not to be confused with general snapshot tests, which compare blobs of text to check for differences.

Where does it fit in your testing strategy?

Visual tests complete your testing strategy by offering a new aspect to check against: the appearance. With little effort, they provide a reliable check for how things actually look in the browser.

More coverage for less work

Let’s say you’re a developer on an online shopping product. You’re tasked with verifying how data flows from the backend to the frontend for a critical product detail page. In particular, you must test the all-important “Buy Now” button.

Using traditional E2E testing methods, you might create a test like the one on the left. But if you’re able to use visual testing, it’s as simple as the test on the right.

The visual test on the right will take an image snapshot of the page, which allows you to verify all of the information tested on the left with a single glance. Not only that, you’re testing that the data flows correctly and that the appearance of the page is correct, too.

This is a dramatic reduction in the work you need to do to write and maintain tests. In the example above, it’s 5x fewer lines of code.

Reuse Playwright tests for visual testing

Because the code above is so minimal, you effectively get visual tests for free. How does that work?

A library sits on top of Playwright’s browser automation and takes snapshots of tested pages. This gives you comprehensive visual coverage of every key page in your app. All without instrumenting a bunch of assertions or maintaining those assertions when your app changes.

There are a couple more handy benefits:

Less work: Visual tests run as part of your existing Playwright E2E test suite. Configuring and doing the setup work for tests can be difficult and time-consuming. With visual tests, you can reuse your test data, mocked modules, and other configurations with any extra code.
Findability: Visual tests are colocated alongside Playwright tests. A snapshot is automatically captured at the end of a test with the option to capture more snapshots at any point mid-test. This simplifies organization and find-ability without switching tools.

How to integrate visual testing in Playwright

Set up E2E visual tests by replacing one line of code. The visual test package, @chromaui/test-archiver, wraps the @playwright/test package to generate snapshots of pages in your Playwright tests.

Now, instead of manually checking your app for visual changes, your workflow can look like this:

Push code to kick off CI jobs (Playwright + visual tests)
Automatically detect visual changes
Review changes to check if they’re regressions or intentional updates
Fix regressions or accept changes as new baselines

Get first access

Our team at Chromatic is building E2E Visual Tests to help you catch UI bugs in Playwright tests automatically. We're currently getting ready for our pilot and there are still some places available. If you’re interested in getting early access (and free usage), fill out the form to see if your company qualifies.

👉 Sign up for early access to Chromatic's E2E visual testing

FAQ

Why not use Playwright’s built-in screenshot feature?
Visual testing is a multi-step process. Playwright’s screenshots are one part of the process. To create a visual testing workflow that devs actually want to use, you’ll also need to store those screenshots, review them, orchestrate CI to capture them, and integrate with Git for PR checks. We handle all that for you and maintain it over time.

What’s more, Chromatic E2E visual tests capture both screenshot and a web archive of the real page (DOM + assets). This allows you to reproduce how your app renders in a real browser, in the same way your users do. No need to step through the entire E2E test again in order to debug, you can jump straight to the page with visual changes.