If your automated accessibility tests pass but a disabled person can’t complete a purchase, what have you actually tested? You’ve tested whatever rules the tool applies to whatever content you gave it. That information has value, but somewhere along the way a lot of teams have become comfortable treating it as evidence that the feature works for people with disabilities. That’s a much bigger claim than the test results support.

Consider a checkout form. The fields have labels, the buttons have names, and the automated accessibility scan reports no errors. The functional tests also pass. A screen reader user then tries to use the country picker and gets no feedback as they move through the options. They make a mistake in the postal code and submit the form. An error appears on the screen, but nothing announces it or connects it to the field. The user is left trying to figure out why the form won’t submit.

There’s no contradiction here. Today’s automatic testing tools cannot examine those behaviors. The fact that everything passed tells us that we haven’t written enough of the right tests.

What your automated scan actually tests

I’m a strong proponent of automated testing because having people repeatedly inspect things a machine can check is a waste of time. But the usefulness of automation depends on what we ask it to do. We need to put more effort into testing the interactions people depend on to accomplish something. A typical accessibility scan examines the page in its current state. It can identify a missing accessible name or an invalid ARIA attribute. But, it can’t establish that focus returns to the correct place when a dialog closes unless something opens the dialog, closes it, and checks where focus went. Similarly, finding aria-expanded="false" on a button tells us very little about whether that value changes when the button is activated. The behavior needs to be tested.

There’s also the matter of what the scanner gets to examine. If the validation errors only appear after someone submits the form, a scan performed before submission won’t examine them. If the dialog’s contents don’t exist until it opens, they won’t be examined either. Running a scanner during a workflow helps with this, but we still need something to perform the workflow and verify the interactions.

Use case testing already addresses this

We already have a methodology for doing that. Use case testing describes a task and asks a tester to perform it using the relevant input methods and (in the case of use case testing for accessibility) target assistive technologies. The task might be registering an account, finding a store, or completing a purchase. The tester follows the steps and records what worked, what failed, and what required a workaround.

One of the things I like about use case testing is that the findings have an obvious relationship to the purpose of the website. A product owner doesn’t need to understand ARIA to understand that someone can’t select a shipping address. A developer gets a specific sequence of actions to reproduce. The test describes something the user actually needs to do, which makes its importance considerably easier to explain.

The problem is repeating it. Manual testing takes time, and the time required grows with the number of workflows and variations you need to cover. A team may do a thorough job during an audit and verify every fix. That doesn’t tell them whether the same behaviors still work after the next three months of development. Without ongoing testing, the only thing preventing a regression is the hope that nobody breaks anything.

Your functional tests may be skipping the accessibility requirements

Development teams already address this problem for other functionality through automated browser tests. Those tests can complete entire workflows, but whether they tell us anything useful about accessibility depends on how they’re written. For instance:

await page.locator('#country-select .dropdown-toggle').click();
await page.locator('li[data-value="CA"]').click();
await page.locator('.btn-primary').click();

That code finds some elements and clicks them. It doesn’t check whether the country picker has an accessible name, whether the options can be operated with a keyboard, or whether their state is communicated to assistive technology. A successful result may tell us that Canada was selected and the form submitted. The accessibility requirements remain entirely untested.

Of course, developers can write tests that check these things. There’s nothing inherently inaccessible about browser automation. The challenge is getting the accessibility requirements into those tests accurately and consistently. The person who understands the requirements may not know the test framework, and the person who knows the test framework may not understand the accessibility requirements.

Meanwhile, we have a manual test script describing what the user should be able to do. It seems unnecessarily wasteful to require someone to translate that script into a separate set of tests and then maintain both.

Make the use case executable

I envision a situation where the use case itself become executable. We already see this, in many ways, in Gherkin Syntax. Gherkin allows you to define interactions in a human-readable DSL that defines the interactions in a way that can then be used for BDD, manual testing, and QA automation.

Here at AFixt, we already have our own design pattern for writing manual use case tests in a way that allows us to surface issues in the process. The vocabulary is already fairly predictable. Locate a control. Move focus to it. Enter a value. Select an option. Activate a button. Verify the result. Each action applies to something with a role and a name: the field labeled “Postal code,” the button named “Place order,” or the dialog named “Delete project?”

With a sufficiently precise vocabulary, the same script could be read by a tester and executed by a tool. The accessibility specialist could write the requirements directly. The QA analyst could add variations. The developer could run the tests during development, and the team could run them again when changes are submitted. For this to be useful, however, the language and the tool would need to enforce some discipline.

First, controls should be identified by their role and accessible name. If a step refers to the button named “Place order,” the test should require that role and name. Allowing the author to fall back to .btn-primary because the semantic lookup failed would defeat the purpose. A missing role or name is something we want the test to expose. In existing functional testing suites, you lose an opportunity to surface an accessibility problem. Using role and name also makes the tests less dependent on irrelevant changes. Changing a CSS class or adding a wrapper around a button shouldn’t require updating a test of whether someone can place an order. Changing the button’s name might require an update, but that’s a change to the interface people use.

Second, the actions need meaningful assertions. Calling a method that requests focus doesn’t establish that focus arrived at the intended element. It also doesn’t establish that the element can be reached by keyboard. Those are separate things, and the test needs to check the behavior it claims to test. The same applies to state changes. A test for an expandable section should activate the control and verify that the content appears and the expanded state updates. It should then close the section and verify the reverse. Otherwise, a control that opens correctly but never closes could pass. A validation test should establish that the field is identified as invalid, that its error message is associated with it, and that the user receives the intended feedback. Simply finding the error text somewhere in the document isn’t sufficient.

Third, the tool needs to reject anything it cannot understand or execute. An unsupported instruction must not be quietly ignored. Neither should a typo cause half an assertion to disappear while the test reports a pass. I’d much rather have a tool tell me that my test is invalid than give me confidence in a check it never performed.

These requirements also benefit manual testing. “Verify that the dialog is accessible” leaves a lot to the tester’s interpretation. A script that specifies exactly where focus should go, how the controls should be reached, how the dialog should close, and where focus should return is much more useful. Making the script executable requires us to be clear about what we expect.

Test what happens when things go wrong

Once those expectations are defined, we can reuse them across variations of a task. A registration form needs tests for missing information, an existing email address, and a password mismatch, as well as successful registration. Most of those cases share the same steps. Copying the entire workflow into a separate file for every variation creates maintenance work and opportunities for inconsistency. The common steps should be reusable, with each variation supplying its own inputs and expected results. This is extremely important for testing accessibility, because error handling is part of using a form. People make mistakes. A user who can submit perfect information but cannot identify and correct a problem hasn’t been adequately served by our testing.

The workflows also give us better opportunities to use the scanner we already have. Open the dialog, then scan it. Submit incorrect information, then scan the resulting state. Perform a search, then scan the results. We can reach the content that an initial page scan would miss and apply the same automated rules to it. We can also repeat applicable checks across controls and conditions. If there are six expandable sections, check all six. Run the task using keyboard input. Run it at a narrow viewport and check whether the layout changes interfere with operation. Automation makes this repetition more affordable, provided we take the time to define the cases.

Be clear about what still requires a person

There are still likely to be limits to what each check establishes. Inspecting a live region’s content doesn’t tell us what a screen reader actually announced. Testing speech output requires access to that output. Similarly, locating a control by role and name doesn’t establish that its instructions make sense or that the overall task is easy to complete. We still need expert review and testing with disabled users. The results should make clear what was checked and what remains for a person to evaluate.

The practical benefit is that we can spend less human effort repeating checks whose expected results are already known. Manual and automated findings can refer to the same use case and the same steps. A tester can investigate the parts that require judgment while automation verifies the behaviors that can be reliably checked.

Keep fixed problems fixed

This also gives accessibility requirements somewhere useful to live before an audit. A use case can be written while the feature is being designed. The expected keyboard behavior, focus management, and error handling can become acceptance criteria. The team can test against those expectations as the feature is built.

Most importantly, the tests can help keep fixed problems fixed. When an audit identifies a dialog that loses focus, the corrected behavior should become a regression test. If a later change breaks it, the developer should get that feedback while working on the change. Waiting until the next audit means giving the defect time to affect users and requiring someone to investigate it all over again.

The value here is straightforward: take the accessibility knowledge already captured in use case testing and make more of it repeatable. Give the people who understand the requirements a way to author them, and give the development team a way to verify them continuously.

If a user can’t finish checkout, a report showing zero automated violations doesn’t help them. We need tests that would have caught the reason they couldn’t finish.

What we’re working on

We will soon begin working on putting this approach into practice: taking accessibility use cases written in a form people can read and turning them into repeatable tests of the interactions those use cases describe. I want the person who understands the accessibility requirement to be able to write the test, and I want the development team to run it whenever they change the feature.

There’s work left to do before we have something ready to release. The requirements described here are what we’re building toward, and I’ll share more as that work progresses.

The reason for doing this is straightforward. We already spend considerable time figuring out where an interaction fails, explaining why it fails, and verifying that someone fixed it. That work should continue to be useful after the audit is over. Turning those findings into tests that catch the problem when it comes back would make a substantial difference for the teams maintaining the software and the people trying to use it.