Forem: Mark Noonan

Open Source Accessibility Plugins in Cypress

Mark Noonan — Wed, 19 Feb 2025 16:02:45 +0000

Last year saw the GA launch of Cypress's commercial accessibility testing solution built into Cypress Cloud. We're delighted to see this helping teams speed up and scale their accessibility efforts.

But accessibility testing itself is nothing new in Cypress, and there are many different ways to account for accessibility in test automation which all complement each other. We’ve blogged in the past about considering accessibility in component tests, and recently added an accessibility testing guide to the open-source Cypress App docs.

When drafting that guide I found I wanted to go a bit deeper into the community plugin side of accessibility and Cypress, as I realized that in 2024 alone, there were multiple accessibility related plugins released, some directly related to testing, and others more geared towards customization of Cypress itself.

In this post we’ll take a look at three recently-released accessibility-related Cypress Plugins: Wick A11y, Cypress Runner Themes, and Cypress Voice Plugin, and hear from their creators.

Cypress Voice Plugin

This plugin was created by Dennis Bergiven, who told me "The voice plugin was inspired by your chat with Filip Hric, where you chatted over various ideas and topics using Cypress". This was a really fun live stream Filip and I did back in 2023, and here's the relevant clip called "How to make Cypress talk". In it, I show how to use built-in browser Speech Synthesis to have Cypress verbally announce something at the end of a spec's execution, especially where the spec might contain many tests or do something that takes some time and you have already moved your screen back to the next code change. (In addition to being a great testing educator, Filip is also the developer of the popular Cypress API testing plugin.)

This was just a proof of concept, as we discussed various ways to hack different functionality into Cypress to customize it for specific testing needs. Dennis described later seeing a practical problem at work that this could help with, saying "The idea came back when some colleagues at work expressed they wanted a faster way to detect a test result when running in the UI."

The end result is a highly configurable voice announcer:

Dennis added that "using native web speech API within the plugin fits right into the Cypress framework as it lives in the browser as well", which made the implementation possible.

Cypress Runner Themes

David Ingraham recognized two needs in the community and created a plugin to solve both. First he created a more color-blind friendly theme for the Cypress command in the Cypress App, to avoid using red and green to hint at the passing/failing status of tests. Then he rolled this up into a general Cypress theming package to combine and update some existing theming solutions like cypress-light-theme and cypress-dark.

David said, "I created cypress-runner-themes after a user on the discord mentioned they were having trouble reading Cypress results due to being colorblind and I noticed there wasn't any existing plugins out there. In combination with the old plugin cypress-dark being unsupported, I thought it would be a good idea to provide a newer theme plugin with accessibility support."

Cypress does not have built-in theme support, but David has been able to leverage the fact that Cypress executes plugin code in the browser, and runs in the same environment, to make modifications to the Cypress interface. Plugins are a common way that the community can get out ahead and iterate on behavior that might be desirable in the Cypress App, without waiting for it to be implemented into the core Cypress library. Here's an example of one of the themes:

"It was my first community plugin so I also wanted to learn something new while making a real impact on the community," David told us. We look forward to seeing more in the future!

Wick-A11y

Created by Sebastian Clavijo Suero, and named for its John Wick-style ability to identify and remove accessibility issues with extreme prejudice, this summer blockbuster adds some visual, reporting, and voice-announcing dimensions to the cypress-axe plugin.

Before we hear from Sebastian, it's worth backing up for a second to explain the connection between all these tools.

At the root is the open source Axe Core library by Deque Systems, which was released in 2015. Axe Core has over 1 billion downloads and is used in many accessibility testing systems such as Google's Lighthouse, as well as in Cypress Accessibility. (Side note: we were delighted to welcome Dylan Barrell, the creator of Axe Core, to Cypress Conf 2023, where he shared his presentation about Agile Accessible Software Development.)

The cypress-axe plugin by Andy Van Slaars was first published in 2018, providing an easy integration in Cypress allowing testers to add Axe Core checks where they chose at different steps of the user journey in their tests. In our docs, we discuss more about these kinds of in-test accessibility scans work, so I won't repeat that here.

While there are other accessibility checkers that also have Cypress integrations, cypress-axe has been the go-to for most Cypress users.

OK, now we're up to date!

Much like the other plugin authors, Sebastian saw an opportunity for extending beyond the current behavior to solve a problem, and shared that solution with others. I send him over a few questions:

What inspired you to create wick-a11y?

I created Wick-a11y to significantly simplify accessibility testing, which is often neglected despite its crucial role in developing inclusive web applications. Although tools like cypress-axe are easy to set up, they provide raw data that requires additional processing for clear interpretation of the results. Wick-a11y seamlessly integrates with Cypress, delivering an easy-to-use solution to assess accessibility, interact with results directly in the Cypress runner, and effortlessly produce detailed HTML reports with screenshots.

What made Cypress a good platform for this?

Cypress is an excellent platform for wick-a11y due to its seamless integration capabilities with tools like axe-core for comprehensive accessibility testing. Cypress's interactive test runner and user-friendly interface allow for real-time interaction with accessibility results directly on the web page, enhancing the identification and resolution of issues. Additionally, Cypress supports detailed logging, providing testers with an efficient way to document and share accessibility findings.

Tell us about your recent addition of voice feedback

I added the voice feedback feature to wick-a11y, acknowledging that many individuals testing website accessibility are affected by disabilities themselves. By providing auditory cues alongside accessibility violations, this feature enhances the testing experience, empowering those committed to inclusivity to engage without barriers and continue championing a more accessible web for everyone.

Here's an example of a test open in the Cypress App, showing some accessibility issues detected with cypress-axe and highlighted by wick-a11y

There is much more to see, so check out the links below to learn more about adding these kinds of accessibility assertions to your Cypress tests.

Links

What comes next

The Cypress community has created a useful array of plugins to support accessibility testing and other customized extensions to the app. This is only a recent subset of plugins related to accessibility in Cypress - there are plugins for other accessibility testing packages beyond Axe Core, as well as accessibility-minded test locators from Cypress Testing Library, and more.

What I like about these three recently released plugins is that they represent extensions and iterations upon existing ideas to solve new needs. They leverage the hackability that's inherent to the browser-based nature of Cypress to great effect, and they set the baseline for future iterations to continue building.

Do you have a plugin for Cypress? You can submit it to our documentation's plugins list.

Why I rarely use `getByRole`: Testing Library and the first rule of ARIA

Mark Noonan — Sat, 04 May 2024 00:16:34 +0000

Note: I've gotten some feedback from a Testing Library maintainer, Matan Borenkraout, about these ideas. I haven't gotten around to incorporating that feedback yet. He had some good examples where ByText or even ByLabelText might not be able to find some elements due to how the label is created or how the implicit role is determined, so what I'm describing here might not handle all cases nicely. I don't think this blows up the premise I'm describing, but I'll be thinking that feedback over and expect I'll be updating the content below in the future.

Intro

To explain the reasons that I mostly stopped using ByRole locators and started specifying elements, I want to zoom way out and talk about the various pieces of the puzzle. Here's a rough table of contents.

Background
What problems Testing Library is intended to solve
Why ByRole is too vague
What makes something accessible
What roles even are
The first rule of ARIA (Accessible Rich Internet Applications)
Semantic HTML, TypeScript, and using the platform
What we expect developers and testers to know when writing and testing their code
Situations where I still use ByRole
Why I love ByLabelText
The use of Test IDs
Overall conclusions

Since this argument is all in the details, this is a relatively long post, so I'm skipping some background on the underlying ideas of front-end testing and locating elements. If you're new to these topics, I have a 2022 post on CSS-Tricks called Writing Strong Front-end Test Element Locators that starts more from the ground up. But for now, I'm going to assume you have some familiarity with how front-end testing works and are interested in best practices for locating elements.

And of course, feel free to skip ahead to the conclusions if this is all familiar stuff.

Background

Testing Library provides a great set of accessibility-focused tools to locating elements when writing tests. It was created by Kent C Dodds, one of my longtime favorite educators on front-end and testing topics. Since the initial release in 2018 as "React Testing Library", it's grown in popularity beyond its React component testing roots. It now has plugins for all major test runners and JavaScript frameworks, and in 2022 some of its patterns were adopted in Playwright's test runner as the recommended way to locate elements.

I do almost all of my testing in Cypress (and for clarity: I work there). I've added Cypress Testing Library to multiple projects over the years, and it's my go-to for a lot of testing tasks, especially when testing forms, using the ByLabelText locator.

One thing I've noticed over time is that I've mostly stopped using the main workhorse of Testing Library, the ByRole locator. When I wrote some guidance for my team a few years ago, I recommended not using it in most cases. I also mention this as a quick caveat when I teach testing or accessibility topics (for example, it comes up in passing in a blog post about user-focused locators that I wrote last year).

I haven't given it much thought over the years, it's just one tool in the Testing Library toolbox and it has its place. My recommendation was driven by what I expect ByRole to accomplish. But recently I've noticed many strong recommendations to use ByRole as the best way to find an element in a test, citing the reason that this helps ensure the underlying application being tested is accessible.

The short version

My argument in this post will be that getByRole is not strict enough to ensure that a given accessible role is correctly implemented. Locating elements this way can create misleading scenarios where your tests imply that an element is accessible, even if it's not. We'll get into more detail on that below, but first I want to give you the TLDR on what I do instead.

The recommended Testing Library approach to find and test a button is this:

screen.getByRole('button', { name: /submit/i }).click()

What I prefer to do is locate an element by a combination of two things that, if they are changed, should fail a test:

The element's specific tag name - button or a, for example
The element's accessible name - "Save" or "Home", for example

In order to do this in Testing Library, for buttons and links, I would use the ByText locator, like this:

// get a "Submit" button
screen.getByText('Submit', { selector: 'button' })

// get a "Home" link in the navigation element
screen.getByText('Home', { selector: 'nav a' })

I do something similar using the following format of the contains command in Cypress. I tend to only look for another locator when this isn't going to get me the element I need:

// get a "Submit" button
cy.contains('button', 'Submit')

// get a "Home" link in the navigation element
cy.contains('nav a', 'Home')

The long version

The rest of this post is really just me trying to push at this from a few different angles and think everything through. I often find I have some opinion that I think is right, and then when I try to write about it, I realize I don't know the exact source of why I think something, or what things might just be assumptions. So it helps to document all the pieces, and it lets somebody else cross-check the ideas.

What is Testing Library for?

The basic premise of Testing Library is summed up by this quote from the home page:

The more your tests resemble the way your software is used,
the more confidence they can give you.

The problems Testing Library solves are described in two ways. The first is about avoiding testing implementation details to have better confidence that test failures matter:

You want tests for your UI that avoid including implementation details and rather focus on making your tests give you the confidence for which they are intended.

The second is about how not testing implementation details makes refactoring less painful:

You want your tests to be maintainable so refactors (changes to implementation but not functionality) don't break your tests and slow you and your team down.

The issue of test stability over time, and avoiding testing "implementation details", had previously often been addressed with a Test ID based approach, which maximizes the stability of tests. Testing Library's authors make the argument that using [data-testid="login"] style locators ignores important user-facing aspects of the elements being tested.

This is true. Test IDs intentionally and explicitly ignore the nature of the elements being used. We'll talk more about them at the end -- there's no disagreement there.

My issue with ByRole locators is the same as the problem with Test IDs, just in a narrower area. If either of these are your only locators and no test ever specifies the specific elements you expect, developers have too much wiggle room to break accessibility while keeping tests green.

`ByRole` locators are too forgiving for safe refactoring

The Testing Library query documentation says this about the ByRole locator:

This should be your top preference for just about everything. There's not much you can't get with this (if you can't, it's possible your UI is inaccessible).

The Playwright documentation puts it in similar terms:

We recommend prioritizing role locators to locate elements, as it is the closest way to how users and assistive technology perceive the page.

Both give examples of doing something like this:

screen.getByRole('button', { name: /submit/i }).click()

The underlying idea I take from this that if an element passes the "role check", it's likely to be accessible: locating an element by its role is intended to achieve the goal of safe refactoring - where "changes to implementation but not functionality" do not break your tests.

However, even though a role is part of an element's "identity" in an accessibility sense, it's not enough. We could refactor an accessible component in a way that fully breaks all accessibility, but still have a green test that uses getByRole. Many implementation changes that leave role intact are actually critical changes in functionality.

Here's an example of this kind of refactoring:

What makes a button accessible?

HTML elements all have implicit roles that describe the nature of the element to people with disabilities, accessing the page using assistive technology. Buttons have the button role, anchor tags with href attributes have the link role, input elements with type text have a textbox role and so on.

When it comes to interactive elements, these roles are not usually useful by themselves. What is useful about them is that they represent an overall contract that element will fulfill. To use the button element as an example, here is at least a partial list of the accessibility "contract" that browser vendors fulfill for a button:

It has the implicit role of button so that users know what it is - a screen reader, for example, would announce it as a button
It has a default browser button style to visually distinguish it from surrounding content
It is placed in the tab order of the document and can be focused with the keyboard
When focused it receives a focus outline
It can be activated with a mouse click, or the enter key, or the space bar
If placed inside a form element, activating the button by any of these methods will submit the form (because its default type is submit)
It will receive specific styling, customizable by the user, in Windows High Contrast Mode (used by people with certain visual disabilities)
It has default browser styles for hover and "pressed" states

That's a lot of behavior that the browser implements for a humble button. The browser also manages this button's presence in the Accessibility Tree, which is how assistive technology like a screen reader can announce the button's existence, what it does, and how the user understands what actions are possible.

And here's what may be surprising: the browser only does this stuff for real button elements. Adding role="button" to a div or a span does not trigger any of this in the browser. If role="button" is used on say, a div with an onClick handler, the entire contract of a button must be implemented by the developer, using extra HTML attributes, CSS rules, and JavaScript listeners. And even then, there will still be gaps - especially in Windows High Contrast Mode, which ignores role="button".

This is the main reason I avoid ByRole locators most of the time. Let's look at this locator again:

screen.getByRole('button', { name: /submit/i }).click()

This will absolutely find a true button element, because of its implicit role, but it will also have no problem finding a div with role="button".

Since these two elements can be very different from an accessibility point of view, it does not seem at all safe to depend on ByRole locators during a UI refactor. Accessibility can change radically depending on the underlying HTML, CSS, and JavaScript, even when roles are preserved during what is intended to be passive refactoring.

So, in the cases where there is a semantic HTML element available, I prefer to just specify that element, and that way the tests will fail if somebody refactors to a pattern that uses non-semantic elements to recreate existing HTML functionality.

This element-level approach also nudges developers towards accessibility success, by forcing them to either follow the first rule of ARIA in their refactoring, or modify the test code if they change the implementation. That means that in a pull request, for example, a reviewer would see unexpected test changes resulting from a refactor (where tests shouldn't change) and hopefully ask why semantic HTML was no longer being used. There's at least an extra layer of protection there.

The first rule of ARIA

ARIA stands for Accessible Rich Internet Applications and it's a specification that teaches us how to create online experiences that are likely to be functional for people with disabilities.

The first rule of ARIA is:

If you can use a native HTML element HTML51 or attribute with the semantics and behavior you require already built in, instead of re-purposing an element and adding an ARIA role, state or property to make it accessible, then do so.

The reason for this is to take advantage of what the browser provides, which is highly likely to work correctly with assistive technology, instead of doing a custom implementation of the same functionality.

One way to think about this is: the absolute best you can do for users when implementing a custom button, is to faithfully recreate everything the browser already does in the "button" contract, except your version won't support High Contrast Mode.

There is no value to writing and owning code that reimplements browser functionality - your company doesn't need it, you don't need it, and your users definitely don't need it.

Related practices

A lot of Testing Library advocates are also fans of other good practices, like using explicit types with TypeScript.

A ByRole locator essentially says that there is only one required property of an element, and everything else about its accessibility contract is optional - the test will accept whatever has that one single property. Specifying the correct semantic element, on the other hand, is a way to force the test to only except something where the browser is implementing the full contract. Following the first rule of ARIA in our tests, as well as our code, gives us the most "type safety" in this sense.

The alternative solution is to assert all of the expected accessibility behaviors in your tests, in addition to locating the element by its role, which is absolutely the way to go if you own the code that implements that functionality.

Another good practice is in web development is "using the platform", not fighting it. The first rule of ARIA perfectly aligns with this: using correct semantic elements allows browsers and operating systems to meet the needs of disabled users, and provides the best compatibility with assistive technology. When people say the web is "accessible by default", semantic HTML is at the heart of this statement.

Developers can still override things and break semantic elements of course, but semantic HTML is still a better foundation and more likely to be accessible. So having our tests depend on this aspect of code seems worth it.

What we expect developers to know

Because of the first rule of ARIA, I don't see any valid use case for role="button" in code. We should only see the role attribute being used for custom widgets that do not have native HTML equivalents.

The Testing Library docs point out something similar to the first rule of ARIA, but don't quite make the same conclusion:

Please note that setting a role and/or aria-* attribute that matches the implicit ARIA semantics is unnecessary and is not recommended as these properties are already set by the browser

This recommendation makes clear that it is expected and desired that developers know they should prefer maintaining the native semantics of HTML. Since they need this knowledge to choose the correct accessible implementation of a given role, we should take the mystery out of it and just specify the element that should be there in our tests. Then there is no need to hope that future developers know all the same nuances about accessibility. They can learn them when their non-semantic HTML causes a test failure.

When I still use `ByRole`

I'll take a ByRole selector any time for roles like menu, menuitem, tabpanel, treegrid... there are lots of useful accessible roles that are not yet implemented in standard HTML. Selecting them by role is a great idea, especially as it allow you to refactor into semantic HTML without breaking a test, when elements with those roles do make their way into browsers.

There are also some rare cases where ARIA roles are needed, even when there is overlap with semantic HTML elements. This might happen when working around some specific browser bug with the native element, for example, where owning the implementation of the accessibility contract is worth it because it lets you do something you can't do otherwise.

Why I love `ByLabelText`

OK, enough about why ByRole is not my thing. Let's talk about some good stuff. Here's my favorite way to locate a form field in Cypress:

cy.findByLabelText('Username').type('myusername')

This is not built in to Cypress -- I use the Testing Library plugin. It's elegant, readable, concise, and requires form fields to have a specific programmatically associated label present, which is also needed for assistive technology.

If I wasn't using Testing Library, I might define a custom command that knows how to find labels in my application, executing something like this:

cy.contains('Username').next('input').type('my-username')

But this depends on knowing the exact DOM structure, and here we don't care about that. There are a lot of correct ways to label a form input, it doesn't matter too much which way is used. One good thing is that here we are specifying a real input element, but Testing Library can do that too if we want to, using the selector option:

cy.findByLabelText('Username', {
    selector: 'input'
  }).type('myusername')

This option is explained in the docs:

If it is important that you query a specific element (e.g. an ) you can provide a selector in the options.

This is also presumably why ByText also takes a selector option. Even though we want to avoid testing implementation details when they don't matter, Testing Library has built-in options so that you can still specify them when they do matter.

In general the risk of developers creating custom inputs built out of divs and spans is pretty low, so I don't do that very often in practice. But it might be useful to use this pattern at the component test level, to give the developers instant feedback if they decide to get creative and reimplement an input out of a custom div.

So why don't we pass this selector option to our ByRole locators, where the risks of using non-semantic elements are higher? Primarily because ByRole doesn't support a selector option. That's for a pretty good reason as far as I can tell, since ByRole is specifically intended to avoid specifying the implementation.

It would be pretty pointless and circular to write code that looked like this:

getByRole('button', {
  name: /submit/i,
  selector: 'button' // ⚠️ not possible, just an example
}).click()

In the above format, the selector property would just override the role. On the other hand, in the ByLabelText scenario, since many elements can have similar labels, passing a selector option to further narrow what is acceptable makes more sense. You could have a Shipping and a Billing form on the same page with the same labels. You could have a select and an input with the same label text.

Or you might want to pass something like a [data-testid="header"] input selector if there are two instances of similar forms on the same page.

Oh yeah, what about `data-testid`?

As a big believer in accessibility, and a longtime user of Testing Library, I want to take a second to stick up for the good old Test ID pattern, which still has its place and can be used in conjunction with specifying the element, or the role, or anything else.

I strongly believe you need to cover the accessibility of your app with automated tests, and that means specifying the parts of the DOM that matter. The HTML we ship to users is our product and we should all pay attention to the actual code we ship. There is an enormous, expensive, complicated system of servers, databases, and various tubes often involved in sending it from us to our users. We might as well send the right HTML.

How I feel about Test IDs is this: if you've already specified the correct HTML output in one place, for example in component tests, then it's fine to use Test IDs elsewhere if you want.

Especially if you have a separate QA team, not as familiar with what the "right" HTML and roles are, we don't need to block them from workflows using generic helper locators, as long as there are tests owned and run by the developers writing code that will fail if HTML accessibility is broken. You must make sure your something in your system will fail if accessibility regressions happen.

On a personal level I'd be happy if everything went red when accessibility broke, because it's a critical thing for me. But as long as accessibility is specified, tested, and owned, then testing some workflows using Test IDs might actually have some upsides for certain teams.

It's also the case that many applications are not built in an accessible way where roles, or even elements themselves, are appropriate for their functionality. Those apps still need to be tested, and if it's not easy or possible to remediate things on the fly, a generic Test ID is a better choice than something that, say, specifies the current wrong role that something happens to have, and then causes confusion.

Test IDs also come in handy when disambiguating different instances of similar elements, and in a few other situations. It's just important to know what they do and do not cover.

I'm hoping to make a PR to the Cypress Best Practices documentation, which already endorses both Test IDs and Testing Library, to get into some of this kind of detail about how to choose the approach. But as you can see, what I've got here currently long and unwieldy. I'm hoping writing this post will help me form a much shorter proposed update to the docs.

If you've ready this far: thanks! And please let me know if you have thoughts or feedback. Am I missing something? Is the ByRole locator awesome in a way I'm not accounting for?

Conclusion

Here's the understanding I'm working with at the moment:

Semantic HTML, when available, is always preferred over custom role attributes for accessibility. This is the first rule of ARIA.
While users do perceive the role of an element, what they they depend on is its entire contract, so a ensuring a role exists is not enough to guarantee behavior doesn't change in refactors.
This means we should not use a ByRole locator if the role we are selecting on can be fulfilled by a native HTML element.
Instead we should specify the HTML element itself in our tests. It will help developers do the right thing, and provide better feedback when refactoring, giving you one extra layer of resistance to accessibility bugs.
We can do this in Testing Library by passing a selector option to our ByText and ByLabelText locators. And we can likely do something like it in our framework of choice even if Testing Library is not available, such as with cy.contains('selector', 'text').

Writing tests this way aligns your tests and your development practices with the first rule of ARIA, which will give you a great foundation for building accessible apps and avoid some common accessibility pitfalls that trap even the most well-intentioned of us.

Last note: Testing Library is still great and I'll still add it to every project to augment whatever I'm doing.

Forem: Mark Noonan

Open Source Accessibility Plugins in Cypress

Cypress Voice Plugin

Links

Cypress Runner Themes

Links

Wick-A11y

What comes next

Why I rarely use `getByRole`: Testing Library and the first rule of ARIA

Intro

Background

The short version

The long version

What is Testing Library for?

ByRole locators are too forgiving for safe refactoring

What makes a button accessible?

The first rule of ARIA

Related practices

What we expect developers to know

When I still use ByRole

Why I love ByLabelText

Oh yeah, what about data-testid?

Conclusion

`ByRole` locators are too forgiving for safe refactoring

When I still use `ByRole`

Why I love `ByLabelText`

Oh yeah, what about `data-testid`?