The reality? Manual testing, $24,000 for 2% vs Automation at $500 for 40%

08 Nov 2021 | Accessibility

Adam Turner
  • Tweet this item
  • share this item on Linkedin

The reality? Manual testing, $24,000 for 2% vs Automation at $500 for 40%

In a series of articles, this being the first, we are going to try to look objectively at the varying capabilities and how to bring the best out of each.

There is ongoing debate about the merits of manual versus automated website testing, and which is the best approach for ensuring website accessibility. There is always scepticism of the opinions, especially when the voices come from either a manual or automated testing company.

It is fair to say that automation can’t test everything, for example a computer cannot know whether the alt text correctly describes the image.

However, it is far too simplistic to say that manual testing is the best approach as automated testing can’t check everything.

All too often, those focused on manual testing (the providers of such services) run down automated testing, but let’s explore the reality. If you were to carry out manual testing properly, is it practical, in terms of time and cost, to check more than a handful of pages? Yet automated testing can look at say 40% of the criteria, although we would suggest the new browser integration Sitemorse has pushes this upwards of 60%. On a typical website of say 480 pages, will manual testing ever be able to practically assess more than a few % (in which case automated testing is 10x or even 20x more effective?).

Also we need to consider the time lag here, where automation could deliver initial results ‘in minutes’ manual testing could take over a month – perhaps value of time saved and being able spend that time focused on improving, addressing the issues found, is also a consideration.

Let’s look more closely at the financial perspective and bring that to the debate:  we’ve made a bold statement as to the cost of manual testing as being between $1,000 and $4,000 per page.

Does between $1,000 and $4,000 per page sound sensible? (There is a cost calculator online from a company called Usablenet https://info.usablenet.com/accessibility-remediation-calculator) Let’s cross check it.

There are about 80 success criteria to check, which is probably the way around you would check from a manual standpoint rather than looking at each individual technique.

Let’s say you could quickly eliminate half of those because the page does not contain anything appropriate; for example, if the page contains no video or audio then you can ignore the whole section on time-based media.

That still leave a lot of stuff to check and some of it will be quite time consuming, such as:

  • keyboard control
  • filling in forms
  • evaluating whether the navigation make sense
  • establishing whether the headings appropriate.

On average, to assess 40 things in a day leaves less than 15 minutes per item. I think it would take more than a day to do one typical page and I think you would be doing well to get an average page assessed in two days.

If the page has a load of JavaScript that pulls in content from the server and content on the page changes depending on what you do, it might take much longer.

So, $1,000 to complete two days’ work on an average page is $500 per day. If a page is four times as complex (and there are plenty of pages out there that are very complex), then a whole week to manually test a complex page is also reasonable.

So, yes, it sounds sensible.

However, I think if you are going to do manual testing, I suggest it would be foolhardy to go through each success criterion. I think manual testing is more likely to test usability, that is, perform the steps of a scenario, such as, perform a search or buy a product, enter a support request, etc.

If you are doing it this, more sensible, way, then first you need to spend time with the client defining the scenarios and the expected test outcomes, then you need to do the test and write up the results. Now it really starts to get expensive!

And that is the stark reality – to manually test an entire site is an expensive process, if you want to do the job properly, and why would you not? Then there’s the issue of new content, content changes etc.

Ultimately, how do you solve this issue? - one conclusion could be, to ensure that you are specific in your requirements and that these are reflected contractually with your suppliers. You need to know you can trust them both now and, in the future, and what will happen if they fail, after all website accessibility is ultimately your responsibility in the eyes of the law and your customers.

In our next article we look at how to bring together manual and automated testing – for instance, automation finds image, checks for alt text (runs intelligent comparison) and then prompts for a human check.