Using automated tools to check accessibility

Post by Mark and Roland, Corporate Digital Team, Welsh Government

Maximising the accessibility of GOV.WALES is a crucial part of providing a great service for everyone. It’s also the law.

But it’s a big job to regularly check the over 46,000 pages which we in the Corporate Digital Team directly oversee on our main website (www.gov.wales / www.llyw.cymru). It’s the sort of task we need software to help with and while automated testing is not perfect, it’s a great way to get good reassurance we are complying with standards. It also leaves us more time to work on those things which need some human attention.

Choosing a tool

There are lots of accessibility testing products. We reviewed some of these products and the experiences of other organisations including:

We chose SortSite Professional because it offers good value with effective reporting at a reasonable cost.

How we use SortSite

An interaction designer runs a scan using a desktop version of SortSite Professional. A different Digital Data and Technology role could do this; however, it works well in our team because our interaction designer has the right mix of skills to interpret the reports.

They sort issues into 2 categories:

development, these are issues which may need some software development to fix
content, these are issues an editor with Content Management System (CMS) access can fix

The interaction designer proposes work to fix development issues. They record this work as tasks for review by colleagues which are then prioritised and assigned to a development sprint. Accessibility and coding standard issues are considered high priority.

Content issues are passed to a content designer. They work with the colleague who originally published the content to agree and make changes to fix the issues. This is a good opportunity to develop skills in our publishing community and identify other activities to avoid similar issues in the future. For example, we may produce new guidance or run workshops to explain how to avoid issues we have experienced.

Once we have acted on all the issues in a scan, we update the GOV.WALES accessibility statement.

Limitations

Automated testing is great for quickly testing lots of content to give some good insight into how we are doing. However, software blindly follows a set of rules and is not a real user. For example, it tells us if alt text is present but it cannot tell us whether the alt text is any good or whether it should be present at all.

SortSite also does not test files. We strive to publish most content as web pages however we still have a lot of files.

To help overcome these limitations, we also complete some manual testing. An example of this is screen reader testing. This testing tends to be more focussed on reused aspects of the site rather than what is published by content editors.

Measurable improvement

A great thing about automated testing is that it gives us data so we can track and report how we’re performing. Pleasingly, we’ve seen a significant reduction in reported issues:

in September 2021 there were 6,938 pages with reported accessibility issues
in July 2024 there were 363 pages with reported accessibility issues

We find very few development issues as they have been fixed following previous automated tests or identified in manual testing before a change is made public. Most issues now tend to be the result of how a specific piece of content has been published, for example, a complex table with nested headers. This illustrates the importance of regular testing as new content is added all the time and it’s crucial to monitor its quality so we can identify problems and intervene quickly.

If you have any questions or comments it would be great to hear from you, leave a comment below or email us at digital@gov.wales.