A few years ago, I gave a presentation titled So You Want an Accessibility Score? about the problems associated with attempts to quantify accessibility into a single numerical score. The topic was not particularly new then, and it certainly isn’t new now. Organizations want accessibility scores for essentially the same reasons they want scores for everything else: scores are easy to understand, easy to compare, and easy to put into reports and dashboards. If you can say that a website is “92% accessible,” that sounds considerably more useful to an executive than handing them an audit containing hundreds of individual findings.

There are legitimate reasons to measure accessibility, and I’ll return to those later. The problem begins when a percentage is treated as a measure of conformance. Saying that a product passed 92% of some collection of accessibility tests is one thing. Saying that the product is therefore “92% conformant” or “92% compliant” is something else entirely.

I’ve argued against this approach for years. Recently this topic came up again in the Accessibility Slack and Grey L Pierce shared some really interesting information that I think adds a lot to this idea of accessibility scoring.

The new Title II rule

On April 24, 2024, the Department of Justice published its final rule concerning accessibility of web content and mobile applications provided by State and local governments. The rule amended the regulations implementing Title II of the Americans with Disabilities Act and, among other things, established WCAG 2.1 Level AA as the technical standard applicable to covered web content and mobile applications. The rule became effective June 24, 2024. (Federal Register)

This rule is important for a number of reasons, but buried within the Department’s discussion of how compliance should be measured is a particularly interesting section for anyone who has ever been asked to provide an “accessibility score”. The Department considered doing exactly that. At 89 FR 31391, the Department states: “The Department considered requiring a certain numerical percentage of conformance to the technical standard.”

The Department considered several possible versions of such a system. The simplest would calculate the number of instances of nonconformance and use that to determine a percentage. More sophisticated approaches could weight failures differently according to such factors as the importance of the affected content, how frequently the content is accessed, or the severity of the accessibility barrier.

These are not unfamiliar ideas. In fact, they represent most of the obvious answers to the problems involved in accessibility scoring. A simple percentage treats all failures equally, which doesn’t make much sense, so the natural response is to create a weighted score. More severe problems receive more weight. Problems occurring in important functionality receive more weight. Perhaps frequently used pages receive more weight than obscure pages. I discuss this in my presentation on scoring.

At first glance, this sounds reasonable. The problem is that none of it answers the question that accessibility requirements are actually intended to answer: can a person with a disability access the content and use the service?

Why percentages don’t describe conformance

The fundamental problem with a percentage-based approach is that accessibility failures aren’t interchangeable units.

Suppose a website is subjected to 1,000 accessibility tests and passes 990 of them. Calling the result “99% accessible” is mathematically straightforward, assuming for the moment that we’ve even agreed upon what constitutes one test. What that number doesn’t tell us is anything meaningful about the ten failures. Those ten failures might consist of relatively minor problems scattered throughout the site. Alternatively, one of them might prevent a blind person from submitting an application for public benefits.

This is not merely a hypothetical example I’ve constructed to make the point. It is essentially the example DOJ used when explaining why it rejected percentage-based conformance. The Department observed that even a very small amount of nonconforming content could prevent access to a government service. A single instance of nonconformance, for example, could prevent a person with a disability from submitting an application for public benefits. (Justia)

This gets to the core problem with percentage-based measures of conformance. The denominator can overwhelm the importance of the numerator. On a sufficiently large website, even a significant number of serious accessibility barriers can represent a very small percentage of the total number of things tested.

The Department consequently concluded: “… the Department continues to believe that a percentage-based approach would not be sufficient to advance the objective of subpart H of this part to ensure equal access to State and local government entities’ web content and mobile apps.” (89 FR 31391 ¶ 891).

That is a much more important statement than simply saying that DOJ couldn’t determine what percentage should constitute a passing grade. The objection is to the underlying premise. A percentage of technical conformance does not necessarily tell us whether people with disabilities have equal access.

There is another problem: what exactly are we counting?

Even if we ignored the problem above, calculating a defensible accessibility percentage is considerably more complicated than it initially appears. If I test a website against WCAG, should I calculate the percentage based on Success Criteria? If 50 Success Criteria apply and the website fails five, is the website 90% conformant? What happens if one of those five failed Success Criteria occurs once and another occurs 5,000 times?

Perhaps we should count individual instances instead. If there are 10,000 images and ten are missing appropriate text alternatives, we could say the site is 99.9% conformant with Success Criterion 1.1.1. But now we’ve created a measurement that gives enormous influence to repetitive content. A site containing thousands of correctly implemented images could make a completely inaccessible application form look statistically insignificant.

We could count pages, but modern web applications don’t necessarily have meaningful “pages” in the traditional sense. We could count components, but then we need a consistent definition of a component. We could count user journeys, but now we’re making judgments about which journeys matter and how they should be weighted.

The obvious response to these problems is weighting. We could assign more weight to severe barriers, important content, frequently accessed content, or critical functionality. DOJ considered those possibilities as well. The difficulty is that a weighted score doesn’t eliminate subjectivity. It formalizes it.

Someone must determine the weights. Someone must decide that one type of failure is worth twice as much as another. Someone must determine what content is important and how much importance should affect the score. Someone must determine whether frequency of use matters more or less than severity.

Once those decisions are made, the resulting number may look extremely precise while being based on a collection of subjective assumptions. The DOJ encountered precisely this problem. The Department noted the difficulty of choosing a particular percentage of conformance or creating a specific conformance formula, particularly given the limited justification commenters provided for how such a system should work. Ultimately: “the Department declines to adopt this approach” 89 FR 31391 ¶ 893. (ADE Commissioner’s Memo)

This does not mean DOJ adopted a simple pass/fail approach

The DOJ’s rejection of percentage-based conformance does not mean that the Department decided that every instance of WCAG nonconformance automatically establishes a violation of the regulation. In fact, DOJ explicitly considered and rejected an approach based strictly on 100 percent conformance.

The final rule instead contains § 35.205, which addresses the effect of noncompliance that has a minimal impact on access. The Department explains that it was persuaded that measuring compliance strictly as 100 percent conformance to WCAG 2.1 Level AA would not be the most prudent approach. The rule therefore provides a limited mechanism under which a public entity may demonstrate that a particular instance of nonconformance has such a minimal effect on access that it does not undermine substantially equivalent access to the same information or functionality. (Legal Information Institute)

Ultimately, the DOJ rejected both ends of what might otherwise appear to be the obvious choices. It did not adopt a rule saying that 95%, 98%, or some other numerical level of WCAG conformance constitutes compliance. It also did not adopt a rule under which every technical failure necessarily means that the entity has failed to comply.

Instead, the Department focused on access. In explaining its chosen approach, DOJ says that it focuses on the experience of people with disabilities using the web content or mobile application. The effect of the nonconformance matters because the purpose of the ADA is not to produce technically perfect websites for their own sake. The purpose is to provide equality of opportunity and full participation. (Legal Information Institute)

I think this distinction is critical to understanding why accessibility scores are problematic as measures of compliance. Ultimately, accessible is conformant.

Conformance and measurement are different things

None of this means that percentages are useless. The problem isn’t measurement. The problem is what we claim the measurement means. Organizations running serious accessibility programs absolutely should collect metrics. They should know how many accessibility defects they’re finding. They should know their severity, know where those defects are introduced, know how long they remain open, know how often previously fixed defects recur, and whether new releases are getting better or worse. They should measure the accessibility of critical user journeys and track their ability to prevent accessibility problems before production.

A percentage can be useful for many of those purposes. If an organization runs the same testing methodology against the same product every month, a change from 70% passing tests to 90% passing tests may be meaningful. It can show progress. It can help identify regression. It can demonstrate whether remediation efforts are working. It can provide management with an easily understood indicator of the direction in which the product is moving. What it cannot do is establish that the product is “90% compliant.”

The distinction here is similar to one I’ve made elsewhere regarding accessibility maturity. An organization can have excellent processes and still produce inaccessible products. Conversely, a particular product can perform reasonably well despite the organization having immature accessibility processes. The measurement can tell us something useful without answering every question we might want it to answer.

An accessibility score should be treated the same way. A score can be an indicator. It can be a KPI and can be a way of comparing results produced by a consistent methodology over time. What it isn’t is a substitute for determining conformance or, more importantly, determining whether people with disabilities actually have access.

The attraction of the accessibility score

I understand why people want a score. Accessibility is complicated, and organizations have a legitimate need to communicate complicated information to people who don’t have the time or expertise to understand hundreds of audit findings. “We have 437 accessibility issues” isn’t particularly useful without additional context. Neither is a spreadsheet listing WCAG Success Criteria. A score promises to solve that communication problem by reducing all of this complexity to a single value.

Unfortunately, simplification has a cost. In this case, the information lost during simplification may be the most important information of all. Two products could receive exactly the same accessibility score while providing dramatically different experiences to users with disabilities. One might contain a large number of relatively low-impact defects while the other contains only a handful of defects that completely prevent users from performing critical tasks. A scoring system can attempt to compensate for this through weighting, but then the score increasingly represents the assumptions of the scoring methodology rather than an objective measure of accessibility.

This was one of the primary points of So You Want an Accessibility Score? A numerical score can look authoritative without necessarily being meaningful. The precision of the number should not be confused with the validity of what the number claims to represent.

DOJ’s discussion at 89 FR 31391 is particularly useful because the Department went through the same exercise at a regulatory level. It considered simple percentages. It considered weighted percentages. It considered the consequences of small numbers of high-impact failures. It considered the difficulty of developing a defensible formula.

Measure accessibility, but be honest about what you’re measuring

There is nothing wrong with telling an executive that accessibility testing results improved by 15% over the previous quarter, provided that the underlying methodology makes that comparison meaningful. There is nothing wrong with building an accessibility dashboard. There is nothing wrong with creating metrics intended to show progress, regression, remediation velocity, testing coverage, or other indicators that help manage an accessibility program. The mistake is crossing the line between saying “this is a useful measure of our accessibility performance” and saying “this is how compliant we are.” Conformance doesn’t work that way, and access certainly doesn’t work that way.

A user who cannot use a website doesn’t receive 99% of the benefit because 99% of the website passed its accessibility tests. The DOJ’s Title II rule recognizes this. Its approach isn’t based on pretending that every technical defect has identical consequences, nor does it attempt to make those consequences disappear inside a percentage. Instead, it asks the more important question: what is the effect of the nonconformance on the ability of people with disabilities to access the information, functionality, service, program, or activity? If a user can’t derive the intended benefit of using the system, it is non-conformant, regardless of how many accessibility tests it passes.

Accessible is conformant.