Automated Accessibility Testing Tool? 9 Questions to Ask

The needs for internet accessibility couldn’t be highlighted better during these COVID-19 times when people with disabilities struggle to handle the pandemic. People shop, continue their education, work remotely, and attend meetings on the Internet now more than ever with no clarity on how long they’ll be in that isolated state.

According to the World Bank statistics on disability, about 15% of the world’s population (currently suffers from some form of disability and represents the world’s largest minority. America has over 61 million people living with a disability (1 in 4 persons), which includes mobility, hearing, visual, and cognitive impairments, all of which affect the web’s use. This is the same America where the Americans with Disabilities Act (ADA) as a non-discrimination law mandates organizations make reasonable accommodations for people with disabilities by providing accessible products and services, including websites and applications. Failure to have an ADA-compliant website could open organizations to lawsuits, financial liabilities, and brand reputation damage. Typically, as a result of non-compliance, web accessibility lawsuits have continued to grow in 2020. 

United States courts have consistently ruled that websites should follow the World Wide Web Consortium’s (W3C) guideline- the Web Content Accessibility Guidelines (WCAG 2.0 Level AA) to be accessible under the ADA. As a technical requirement, WCAG 2.0 AA covers a wide range of recommendations for making web content more accessible to a broader range of people with various kinds of disabilities.

Removing accessibility barriers does not just benefit persons with disabilities but everyone in general. Innovative organizations that want to do the right by providing accessible websites and apps have a lot to gain by delivering disabled access and accommodating this significant portion of the population. These include eliminating the risk of a protracted lawsuit and potential financial loss.

The first stage of making a website accessible is the identification of accessibility issues by auditing web properties. The web accessibility audit tests against technical requirements such as the WCAG 2.0 AA success criteria using website accessibility testing tools and manual/functional testing to discover barriers faced by people with disabilities. Automated website accessibility testing tools are software programs or online services that scan your web content to determine elements that do not meet accessibility guidelines (WCAG, Section 508, etc.). An example is the WAVE accessibility evaluation tool. 

For automated accessibility testing against WCAG 2.0 AA success criteria, how do you rely on the testing tool? What are the essential factors to consider when choosing and what questions do you need to ask?

Website accessibility testing tools are meant to save time and effort during the evaluation, not add to the confusion. In most cases, the testing tools are designed as a separate system from Quality Assurance (QA) and development systems. Hence, it could be handled by a non-technical figure within the organization, which means it must be user-friendly and easily understandable.

You want to be sure the testing tool supports testing against accessibility standards your organization is required to conform with. The majority of accessibility standards now reference the international standard- the Web Content Accessibility Guidelines (WCAG) 2.0. The WCAG 2.0 describes testable success criteria and defines three conformance levels: A (lowest), AA, and AAA (highest). The usual target level is Level AA.

As an upgrade to WCAG 2.0 AA, WCAG 2.1 AA is broader and allows to make web content even more accessible to a more comprehensive range of persons with disabilities. Other than the WCAG standard, testing tools can also accommodate the creation and maintenance of internal custom standards and rule sets.

Modern programming languages (e.g., Java, PHP, JSP) can make a DOM but are not the same as the browser DOM. The DOM that must be tested here is the browser DOM.

Testing the browser DOM is vital in discovering what users are experiencing and in testing accurately. Hence, you want to check for a tool with its own (or uses a) headless browser to test the final browser DOM.

Ideally, accessibility testing should be done earlier in the process, and not after a system is already in production. Hence, the testing tool should be flexible enough to support code-in-development testing. This means the tool should allow users (code developers, designers, and content authors) to test by uploading files directly and entering source code directly without requiring that code to be uploaded to a publicly accessible URL.

Websites with a large number of web pages and several subdomains should be looking for a testing tool that allows for targeted testing or can be configured to test a selected set of pages (out of the tens of thousands).  

More so, it is now common for sites to use templates and content management systems, which means the same code to be reused every time an interface component of a specific type is used on different pages. In this case, any error in templates means every page will have the same error when it is actually one. The testing tool must now report results that separate common issues from unique ones to reduce repetitive reporting and provide more accurate results.

Accessibility experts may not necessarily do automated accessibility testing, and whoever is testing may rely entirely on the recommendation given by the testing tool. A testing tool that provides inaccurate or unreliable results is harmful to accessibility, dangerous for the project budgets and timelines, and ultimately catastrophic to the organization. The result should be useful, detailed with step-by-step, code-level examples of how the accessibility issues can be solved and not be full of jargon.

Tools that poorly interpret the WCAG will confuse the team and probably delay decision-making and time spent on the project.

No automated testing tool can test all accessibility or WCAG issues, so the tool should provide guidance on testing that must be performed manually. The more the tool can help with the manual review, the better. The recommendations provided should be clear and easy to understand for even a non-technical person.

It is common to have tools that can work seamlessly within existing toolsets and workflows rather than have it in a completely separate system. Integration with existing processes is better and more convenient for development, QA, and content teams.

An organization with sites that require frequent updates or adds new content needs this the most since it becomes more challenging to manage accessibility manually. Hence, the need for an automated testing tool that can regularly test the entire site (and can also limit locations to be tested) for accessibility issues. Users should check if the tool can perform scheduled monitoring to ensure periodic testing of the site.

Making a Decision

Automated accessibility testing tools are a must for organizations that are new to accessibility, and they can be quite expensive, depending on the tool and the organization’s specific needs. The testing tool will identify accessibility issues (bugs) caused by not doing it right in the first place (due to lack of knowledge or due process), but they cannot solve those issues. 

Although automated testing tools for ADA compliance tests are helpful, they can only capture around 30% of accessibility issues. Also, they can give false positives; hence, they will require human judgment in some areas. Perhaps you should consider training your staff on accessibility to ensure they can perform various checks and routine audits to identify other barriers that might prevent disabled persons from completing their interaction with the site.

As a precautionary move, it is customary to get training on using the tool and get a functioning test drive of the product to determine its quality. Since the tool is not likely to be cheap, you can have your staff run a real-world use of the tool for some time to be sure it’s fit for purpose and user-friendly.

Large website owners should also consider tools that test more than a single page at a time (especially for post-deployment web pages). They should instead check for testing tools capable of checking multiple types of barriers, extensive scale testing, and the speed at which the tool can test those large numbers of pages or the entire site. It will also be essential to be sure the tool can be integrated into the different work environments- the development, QA, and compliance processes. As an extra, you can also check if the tool allows tests for more than just accessibility, such as data privacy, usability, SEO, and general website performance.

What is most important is making an informed decision on the product that is best suited for your organizational needs. Familiar with some? Kindly share your experience with us in the comment section.