Anyone who performs accessibility audits eventually gets asked the same question: How many pages should we test?
There is no especially satisfying answer to this question. Obviously, testing every page is preferable when that is practical. For a sufficiently small website or application it may be possible to do exactly that. It does not take a very large product, however, before exhaustive manual testing becomes prohibitively expensive and time consuming. At that point, some form of sampling becomes necessary and the question changes from whether to sample to how that sample should be created.
WCAG-EM provides what I think is, overall, a very good answer to that question. Its approach is considerably more sophisticated than simply choosing some arbitrary number of pages. Step 2 requires the evaluator to explore the product and identify, among other things, its common views, essential functionality, different types of content and interaction, different technologies, different templates and coding styles, different authorship, and functional components such as date pickers, modal dialogs, and carousels. It even encourages identification of design systems, content management systems, front-end frameworks, and libraries. Step 3.1 then requires the evaluator to construct a structured sample that represents what was discovered during that exploration.
This is exactly how sampling for an accessibility audit should work. A good sample is not merely a smaller collection of pages. It is a collection intended to represent the meaningful differences that exist within the system being tested. Unfortunately, WCAG-EM then follows this very good approach with what I consider to be a glaring problem.
Step 3.2 requires the evaluator to supplement the structured sample with a randomly selected sample equal to 10% of the structured sample. WCAG-EM says the random sample acts as an indicator that the structured sample is sufficiently representative and that correlation between the findings from the two sampling approaches increases confidence in the overall evaluation. Curiously, WCAG-EM also says that the random sample does not need to be selected according to “strictly scientific criteria.”
I raised an issue about this during development of WCAG-EM 2.0 because I question both the usefulness of this requirement and the assumptions behind it. The more I have thought about the issue, the more convinced I have become that random sampling, at least as described by WCAG-EM, adds little value to most accessibility audits and is based on a model of websites and web-based systems that is increasingly disconnected from how those systems are actually built.
What is the random sample supposed to accomplish?
Random sampling is extremely useful when it is used for the purpose for which random sampling was designed: making an inference about a population from a smaller number of observations. If, for example, we wanted to determine the percentage of web pages containing a particular characteristic, we could take a probability sample and use the results to estimate the prevalence of that characteristic across the larger population. The surprising thing about sample size is how quickly population size stops mattering. Even if we assume that there are tens of trillions of web pages, a simple random sample of approximately 385 independent observations is sufficient for an estimate at a 95% confidence level with a margin of error of approximately five percentage points, assuming the conservative case where the expected proportion is 50%. Approximately 1,068 observations gets the margin of error down to three points, approximately 2,401 gets it to two points, and about 9,604 gets it to one point.
There are, of course, several important assumptions buried in those numbers, including that we actually have an appropriate probability sample and that the observations provide sufficiently independent information. The point is not that accessibility auditors should use those particular sample sizes. The point is that statistical sampling is based on the relationship between the research question, the characteristics of the population, the sampling method, and the degree of uncertainty we are willing to accept. WCAG-EM’s 10% rule has no such basis that I can identify.
If my structured sample contains 30 pages, WCAG-EM requires three random samples. If it contains 80 pages, as in WCAG-EM’s own example, I need eight. If it contains 150 pages, I need fifteen. Nothing about that calculation takes into account whether the underlying product has 500 pages, 50,000 pages, or 50 million pages. It does not take into account any target margin of error, confidence level, expected frequency of failure, or degree of similarity between observations. The random sample size is simply calculated from the size of another sample that was deliberately selected using professional judgment.
That would not necessarily be a problem if there were a compelling non-statistical reason to do it. But this is where the rationale becomes unclear. WCAG-EM says that the random sample is used to determine whether the structured sample is sufficiently representative. Step 4.3 therefore requires the evaluator to determine whether the random sample reveals types of content or outcomes that were not represented by the structured sample. If it does, the structured sample needs additional attention.
In essence, the random sample is being used as a test of whether the evaluator did a good enough job constructing the structured sample. I do not think that is a good reason to make random sampling a methodology requirement.
WCAG-EM already spends a substantial amount of effort describing how the evaluator should explore the product and construct the structured sample. Step 2.3 alone provides a very useful list of characteristics that may indicate meaningful differences between samples. Step 3.1 requires representation of the common views, essential functionality, sample types, technologies, and other relevant content identified during that exploration. WCAG-EM further notes that careful selection can significantly reduce the sample size while maintaining appropriate representation.
If those steps are inadequate to produce a representative sample, the solution should be to improve those steps. Random sampling looks to me like a band-aid for a poorly constructed structured sample.
This was one of the principal points I made in the issue I filed against Step 3.2. I also argued that the value of random sampling depends heavily on why the evaluation is being performed in the first place. That distinction is vital because accessibility testing intended to estimate conformance and accessibility testing intended to inform remediation are related activities, but they are not the same activity.
Compliance measurement is different from remediation
There are perfectly legitimate circumstances in which accessibility testing is primarily a measurement exercise. A government may need to monitor the accessibility of thousands of public-sector websites, a researcher may be interested in comparing accessibility across industries, and a regulator may want a standardized procedure that permits comparisons between different entities. One of the WCAG-EM editors responding to my issue pointed out that sampling is used in the evaluation of thousands of websites and applications monitored by the Dutch government. This type of use case is not difficult to understand.
In those circumstances, however, sampling needs to be designed around the thing being measured. If the purpose is to support statistical inference (in other words: a grade), then the sampling methodology should be statistically defensible. Calling a sample random while simultaneously saying that it need not follow scientific sampling criteria does not provide the benefits that are normally associated with random sampling.
This becomes especially important because WCAG-EM itself is quite careful about what an evaluation of a sample can establish. It notes that, in the majority of cases, use of WCAG-EM alone does not establish a WCAG conformance claim for the entire digital product. This is entirely appropriate because there may be failures outside the evaluated sample. But it also raises the obvious question: if the random sample is not statistically sufficient to permit an inference of conformance across the unevaluated population, what exactly is gained by making it mandatory?
That question becomes more important when the purpose of the audit is not primarily to measure conformance but to identify work that needs to be performed. In more than twenty years of doing accessibility work, the overwhelming majority of audits I have encountered have been performed for this reason. An organization has a website, application, mobile app, or other digital product and wants to know what is wrong with it so that the problems can be fixed. Sometimes that work is driven by litigation or regulation. Sometimes it is driven by procurement requirements or internal policy. Whatever caused the work to begin, the audit usually becomes an input to remediation. In that context, testing effort has a direct cost and the utility of an additional test should be considered in terms of the additional useful information it is likely to produce.
This is where page counts begin to become misleading. Suppose an application uses the same inaccessible date picker on 50 pages. If I test all 50 pages, I can certainly report 50 occurrences of the problem. That information may be useful for communicating impact on the end user, but it does not mean the development team has 50 unrelated defects to remediate. If those 50 instances are all generated by the same DatePicker component, there is going to be one underlying defect and one underlying fix.
On the other hand, consider five different pages, each containing an unrelated accessibility problem caused by a different component or implementation. The raw page count is much smaller, but the remediation problem may be considerably larger. There may be five code changes, five regression risks, five responsible teams, and five separate things that need to be validated after remediation. The page is therefore not necessarily a very useful unit for estimating either the amount of remediation work or how much new information an evaluator obtains by testing another page.
WCAG-EM recognizes this in other parts of the methodology. Step 4.1 specifically discusses repeated components, such as navigation, headers, and search forms, and says that they generally do not need to be evaluated on every occurrence unless they appear or behave differently. The obvious reason is that repeatedly testing the same implementation provides very little additional information. The same principle should influence how the sample is constructed in the first place.
Modern systems make page-based random sampling even less useful
Consider three hypothetical systems.
The first is a 500-page website that has existed for twenty years. Over that period it has had three redesigns, two content management systems, and content contributions from dozens of departments. Some pages use old templates that were never migrated. Others contain hand-authored HTML. Some areas are maintained by the marketing department while others are maintained by human resources, legal, support, or individual business units. The site also contains various third-party tools that have accumulated over the years. On a system like this, random sampling can have some utility because the population itself may contain substantial implementation and authorship diversity. Selecting an unexpected page from an old corner of the site might genuinely reveal something that was overlooked during structured sampling.
Next consider a SaaS application built in React. It has 80 reusable components, 20 primary view types, and several important workflows. Customer records are available at URLs such as /customers/12345 and /customers/67890. Reports can produce thousands of combinations of filters. Users with different permissions see different functionality. Application state determines which controls are displayed. Despite the underlying implementation being comparatively compact, the application may be capable of producing millions of distinguishable URLs or screen states.
How many pages does that application have? More importantly, why would we care? Testing /customers/12345 and /customers/67890 may provide essentially the same accessibility information if both are manifestations of the same CustomerDetail component. On the other hand, a particular customer record may cause a different component to appear because that customer has no transaction history, has an unusually long name, belongs to a different account type, or has some other characteristic that causes materially different output. The reason to test the second customer record is therefore not that it is another page. The reason is that it exposes something different.
Last, consider a large e-commerce system with 5 million product URLs. Those product pages might all be generated from the same product-detail template and the same set of components. If the image gallery has a keyboard accessibility problem, that problem may occur 5 million times. It is unquestionably a serious issue with enormous scope, but evaluating another 100 randomly selected product pages is unlikely to teach us very much about the underlying defect and certainly will do nothing of value when it comes time to remediate the site.
At the same time, this e-commerce system may contain a checkout process consisting of only five views. Those five views might contain address entry, shipping choices, payment controls, validation, error handling, third-party integrations, and order confirmation. An account-management area may include authentication, password recovery, saved payment information, returns, and order history. Those portions of the system account for an almost trivial percentage of its URLs while representing a large share of its functional and accessibility complexity.
A random selection from a population dominated by product pages naturally tends toward product pages. That is not a flaw in random sampling. It is what random sampling is supposed to do. A representative random sample of the page population should be dominated by the things that dominate the page population. The problem is that this may be exactly the opposite of what we want from an accessibility audit intended to inform remediation.
For remediation purposes, we are usually looking for differences. We want the strange template that is used in only one section. We want the complicated interactive control. We want the unusual error condition. We want authentication. We want financial transactions. We want the third-party scheduling system. We want to know whether a different team used a different JavaScript framework. We want the older content created before the current design system existed. We want the parts of the product that create new opportunities for accessibility failures. The rare thing can be more valuable to test than the common thing precisely because it is rare.
WCAG-EM’s structured sampling guidance already understands this. Step 2.5 specifically calls attention to sensitive and high-risk functionality such as authentication, management of personal information, and financial transactions. Step 2.3 tells the evaluator to identify differences in components, templates, authorship, technologies, content types, dynamic behavior, and interaction. Step 3.3 requires complete processes. These are purposive choices, not random ones.
There is also a curious detail in Step 3.2 that illustrates this problem. WCAG-EM says that if the random sampling procedure selects a view that is identical to one already included in the sample, another view should be selected. That is sensible, but that now means we are rejecting a valid random selection because professional judgment tells us that it probably will not add useful information. We then continue sampling until we find something sufficiently different. At that point, professional judgment about implementation diversity is doing the useful work, not randomness.
How many pages is enough?
I do not think there is a responsible way to answer this question with a number or percentage. WCAG-EM itself essentially acknowledges this before arriving at its random-sampling requirement. Its discussion of sample size identifies product size, age, complexity, consistency, technologies, development processes, and other characteristics as factors influencing the amount of testing required. It also recognizes that careful sample selection can significantly reduce the number of samples necessary. That is where I think the answer lies.
A 500-page website assembled by dozens of people over twenty years may require more page samples than an application capable of producing 500,000 URLs from a small and tightly controlled set of components. Conversely, an application with only 30 primary views may require extensive testing if those views contain many different states, interactions, roles, conditional controls, and complex processes. The number we should care about is not simply the number of pages. What matters is how much meaningful variation exists in the system and how well the evaluation sample represents it.
For an audit aimed at remediation, I would go further. The sample should be selected to maximize the likelihood of discovering materially different accessibility problems. That means understanding the templates, components, technologies, authorship, content types, states, user roles, workflows, and other implementation characteristics that can cause accessibility outcomes to differ. It also means deliberately seeking out unusual and high-risk functionality rather than relying on its probability of appearing in a random selection.
This does not mean that randomly exploring a product is useless. An evaluator can absolutely discover something unexpected by wandering into areas that were not initially identified. Random sampling may also be entirely appropriate in research and monitoring contexts where the sampling method is designed around a genuine statistical question. The problem is making a non-scientific random sample equal to 10% of the structured sample a requirement of the methodology.
When I originally raised this issue, I argued that random sampling should be removed until there is meaningful evidence demonstrating what it contributes beyond a properly constructed structured sample. I still believe that is the appropriate burden of proof. Manual accessibility testing is expensive. If a methodology requires additional testing, there should be a defensible reason for believing that the additional work produces information worth the cost.
WCAG-EM’s Step 2 and Step 3.1 already provide an excellent foundation for constructing a representative sample. They recognize nearly all of the things that actually cause one part of a modern digital product to differ meaningfully from another. The methodology could improve those provisions further, particularly by giving greater attention to reusable components, application states, role-dependent functionality, and other characteristics of modern web applications. What it does not need is an arbitrary 10% random sample intended to verify whether the evaluator followed the good guidance that came immediately before it.
If the objective is statistically defensible measurement, use a statistically defensible sampling method. If the objective is to inform remediation, build the sample around the things that are likely to produce different accessibility problems. And if a random selection routinely discovers important portions of the product that were missing from the structured sample, the lesson is not that every audit needs random sampling. The lesson is that the structured sample was not representative enough in the first place.
WCAG-EM 2 is a great methodology for performing accessibility audits that is more or less exactly how I’ve advocated doing them for nearly two decades, but its requirements around random sampling are a mistake that provides no value compared to other, more accurate and relevant approaches.
