Every accessibility program eventually asks the same question. We run automated scans every week, and we commission a manual audit once or twice a year. Is there any relationship between the two? If the scan finds 400 issues on one site and 40 on another, does that tell us anything about what the human auditor will find?
It is a fair question. Automated testing is cheap and repeatable. Manual testing is expensive and the only way to reach a defensible conformance claim. If the cheap signal predicted the expensive one, even partially, you could scope audits more accurately and see risk building between engagements.
The short answer is that a relationship exists. It is just not the one most people expect, and you can only see it if your data is structured in a particular way.
The industry’s favorite number is the wrong question
Most discussion of automated versus manual testing collapses into a single figure: the percentage of accessibility issues automated tools can find. Estimates range from under a third to more than half. The spread comes almost entirely from what is being counted. If you count issue instances, automated tools look strong, because the issues they detect (missing alternative text, empty links, insufficient contrast) tend to repeat hundreds of times across templated pages. If you count WCAG Success Criteria, they look weak, because many criteria have no reliable automated check at all.
Both numbers are true, and neither answers the question a program manager actually has. “What percentage can automation find?” is a statement about tools in general. The useful question is narrower and more practical: For our organization’s properties, which automated signals reliably predict what a human will find, which say nothing, and where are the blind spots?
That question has an answer, but it has to be computed from your data. It cannot be borrowed from a vendor’s marketing slide.
Why raw volume misleads
The intuitive approach is to correlate issue counts: more automated issues should mean more manual failures. In practice, raw volume is a poor predictor, for three reasons:
- Template repetition inflates automated counts. A single defective navigation component on a 2,000-page site produces 2,000 automated findings, but a human auditor records it once, as one issue with one remediation. A site with one bad header can look ten times worse than a site with ten distinct problems.
- Many criteria are structurally invisible to automation. A large number of accessibility issues require human judgment, such as whether instructions rely on shape or location or whether an error message actually helps someone recover. A site can be clean to a scanner and still fail at least two dozen criteria. Leave these criteria in the analysis and they drag any correlation toward zero. Take them out quietly and the correlation is overstated.
- Automated results and manual results aren’t independent. In most audits, the human tester sees the automated results first. That is efficient, but it anchors the tester’s judgment and makes the two data sets agree more than they otherwise would. Any honest analysis has to account for it.
So the answer to “do more automated issues mean more manual failures?” is “sometimes, for some criteria, once you normalize properly.” That is less satisfying than a single number, but it is far more useful.
What actually works: four measurements
Once automated and manual results are structured so they can be lined up, four measurements turn out to be worth having.
1. Agreement per Success Criterion
For each WCAG Success Criterion, within each audit, compare what automation reported with what the human concluded. Over enough audits, this produces a conditional probability per criterion: when the automated rules for 1.1.1 report failures on a page, how often does the manual audit also fail 1.1.1? Some criteria show strong agreement. Contrast is the classic case, because the underlying test is largely computational. Others show weak agreement: automation detects something related, but the human’s judgment depends on context the scanner cannot see. Some show no automated signal at all. The breakdown is the key output of the exercise. It tells you, for each criterion, how much weight to give your weekly scan results between audits.
2. Precision of each rule
Every automated rule can be scored on how often a human confirms what it reports. A rule whose findings are routinely confirmed is one you can act on immediately, without waiting for review. A rule whose findings are frequently dismissed as false positives is one that should either be fixed or demoted to “needs review.” This type of thing is measurable only if dismissals are recorded as structured data rather than silently ignored. A false positive that nobody writes down teaches the system nothing, so it is important that the system facilitate and track them.
3. Recall: where automation is blind
The reverse question is equally important: which manual failures had no corresponding automated finding on the same page and criterion? These are your blind spots. They tell you where scanning will give you false comfort, and where between-audit monitoring needs a human checkpoint or a scripted user-flow test instead. It is also the best input to test rule development: it shows where new automated checks would close the largest gap.
4. Normalized volume as a scoping signal
Volume becomes useful again once it is normalized: counting distinct defects (deduplicated by element fingerprint or component) rather than raw occurrences, and restricting the comparison to criteria where automation has demonstrated agreement. A normalized automated profile can then inform audit scoping. It can’t predict the audit’s outcome, but it can show which components are likely to need the most manual attention and roughly how much effort an engagement will take. To be clear: this is a planning aid, never a conformance determination. Conformance is a claim a qualified human makes and stands behind.
What it takes to answer this question
All of the above measurements depend on the same thing: automated and manual results stored so they can be joined. That is rarer than it sounds. In most organizations:
- Automated results live in a scanning tool, keyed by the tool’s own rule names.
- Manual results live in a PDF report or a spreadsheet, keyed by WCAG Success Criterion and written in prose.
- The two use different identifiers, different severity scales, different notions of “page,” and different granularity.
- False positives are dismissed in the scanner’s interface and never recorded anywhere else.
You can’t correlate data that cannot be joined. Most attempts stall at the spreadsheet stage, mapping rule names to Success Criteria by hand.
Why AFixt is positioned to answer it
AFixt is unusual in this market because we sit on both sides of the question, and we are building the connecting tissue deliberately.
We write our own automated rules. Our team has been building automated accessibility testing since the early 2000s. Our current rule library is our own: each rule maps to specific WCAG Success Criteria, and each is classified by what it can actually decide. Some checks are fully automatic, some are assisted (meaning they find candidates for a human to judge), and some are manual procedures. Most automated tools report everything as a pass or a fail, which is exactly what blurs the line between a reliable signal and noise.
We perform the manual audits ourselves. AFixt’s core business is expert manual auditing and remediation against WCAG 2.2, ADA, Section 508, EN 301 549 and the European Accessibility Act. Our auditors work from the same rule and criterion definitions our automated engine uses. The human verdict and the automated finding therefore describe the same defect in the same vocabulary.
Both kinds of result live in one finding model. Our platform records automated findings and manual evaluations as the same kind of evidence object. Each one is tied to the same property being assessed, the same scope item, the same Success Criterion and the same rule identifier. A finding can be marked as automated, manual, or automated-and-confirmed-by-a-human. When an auditor dismisses an automated finding as a false positive, that decision is recorded as structured data with a reason, not lost in a scanner’s UI.
Humans attest; the evidence is kept. Every conformance determination in our platform carries a named, qualified human attestor, and its evidence is locked once the audit is signed off. That was designed for defensible documentation, and it also leaves a clean history: exactly what the automation said, and exactly what the human concluded.
Put those together and the four measurements above stop being a research project and become queries over data that already exists.
Turning correlation into decisions
Measurements only matter if they change what you do. Here is how we can turn test results into action for an organization’s accessibility program:
Trust the right alerts between audits. For criteria where automation shows strong agreement with manual results, a new automated failure is a real regression and should be triaged immediately. For criteria with weak agreement, the same alert is a prompt for review. Your monitoring should stop treating every finding as equally urgent.
Close the blind spots on purpose. The recall analysis shows which manual failures your automation never sees. Those manual failures should become targets for scripted user-flow tests (checkout, sign-in, account management) run with real keyboard and assistive technology interactions via UCDL, or for a scheduled human spot-check. By doing so, you stop relying on a clean scan as evidence of a clean product.
Scope audits on evidence. A normalized automated profile, read against your past audit history, shows which parts of your system are likely to carry the most manual effort. Proposals and timelines get more accurate, and auditor hours go where the risk is.
Measure your tools, not just your sites. Rule precision tells you which automated checks your developers should act on directly, in CI, without waiting for an expert. It also tells you which checks generate noise that erodes their trust in the whole program.
Track the gap over time. As teams fix the issues automation can see, the remaining risk shifts to the criteria it cannot. A program that reports only automated counts will show steady improvement while its actual conformance stalls. Tracking agreement and blind spots per criterion makes that shift visible before an audit, a complaint or a demand letter does.
The caveats we take seriously
Analysis like this is only credible if its limits are stated plainly.
- Sample size matters. Agreement rates for a single organization with two audits are noisy. The figures become reliable as audit history accumulates, and we report confidence alongside them rather than presenting early numbers as settled.
- Anchoring is real. Where it matters, we can measure it, by comparing audits where the tester did and did not review automated results first.
- Prediction isn’t conformance. No correlation, however strong, replaces a human evaluating a Success Criterion. We use these signals to direct expert attention, never to stand in for it.
The bottom line
There is a relationship between automated and manual accessibility results. It is not a single percentage, and raw issue counts hide it more than they reveal it. It shows up per criterion, per rule and per organization, and only when both kinds of result are recorded in the same structure with the same identifiers.
That structure is what we built. Because AFixt writes the automated rules, performs the manual audits and records both as one body of evidence, we can tell you which of your automated signals to trust, which to treat with caution, where your scanning is blind, and how to plan your next audit around what your own history shows.
If your program has years of scan results in one place and audit reports in another, and you have wondered what the two are telling you together, that is a conversation we would like to have.
