Time to retire the phrase "Automated Testing" and use "Checking" instead

UPDATE to this post, April 18th, 2015: for purposes of automated testing, I'd like to define "check" as a specialization or subclass of "test" where the verifications are limited to those specifically coded or otherwise determined in advance. The usefulness of "check" is limited to what people also call "automated test," and has a business justification that it avoids confusion and risk: it avoids the confusion that a manual test involving a GUI or web page, once automated, obviates the need for a human to run the manual test, and it avoids the quality and business risk that would result from losing the corresponding measurements of product quality.

***

It’s time to retire the phrase “automated testing.”

Given that software testing is about measuring, communicating and promoting quality, leadership often sees automation – that is, making a software product do things automatically –as a way of doing all of the above faster. Unfortunately, it does not work that way.

People are smart, but computers and computing power are not smart. People running user stories or test cases or doing exploratory testing are very good at finding large numbers of bugs, within the limits of attention, getting tired or bored, etc. People are great at spotting things that are not as they should be, e.g., a flicker in an icon over here or a misalignment of a table over there, or a problem of discoverability.

When automated product testing is done well, it has huge value: it is excellent at regressing product quality issues quickly, repeatedly, and tirelessly. Automation does not get tired or bored. Computers are very good at processing numbers and repeating procedures, and doing them fast and reliably, and at e.g. 3AM local time when your people are home sleeping.

However, automation is not good at finding product bugs or anomalous issues like a flickering icon. You need good human testers for that.

Instead of “automated testing,” it’s time to use a term proposed by James Bach and Michael Bolton (see their post http://www.satisfice.com/blog/archives/856 ) to define automation that drives tests: Checking. A single automated procedure that measures a defined aspect of quality for the SUT is a “check.” The term “check” applies where more commonly a professional in the space might use the term “automated test” but since testing is an intelligent activity done by humans, the term “automated test” becomes an oxymoron; once a test is automated, it is no longer a test in the same sense. If done well, it is fast, reliable, tireless and highly repeatable, but the value is very different from the same procedure run by a testing professional.

A skilled and experienced tester running a manual test can discover, characterize and describe as a bug any of a broad range of issues. The range of potential issues found has few limits, and is driven by the intelligence, creativity and observational skill of the tester. However, to take that manual test, automate it, and then run it offline (without human observation or intervention) or in the lab, severely restricts the discoverable range of issues. Automated tests are capable of flagging issues that block the procedure of the test, and issues that are the topic of explicit verifications or metrics coded into the test or automated test harness, but they do not measure anything else about the product. The automated test will usually run faster than the manual test, and a well-written automated test will run more reliably and repetitively than the manual test, but it does not replace the manual test. If the automated test taken from the manual test above runs and passes a thousand times, running the manual version of the test once could still find important issues.

The team therefore still needs the manual test, and in the context of measuring product quality, there is benefit to tracking the manual test and the manual test results separately from the automated test and the automation results. The manual and the automated version of the test both have their values for quality, and one does not replace the other.

It’s time for the industry to use “check” because this term emphasizes that automating a test is not the same as running the test faster and does not obviate the manual version of the test.  The need for manual testers will always be there for the team; however, a well-designed and frequently-run set of checks can make manual testing faster, more effective, and more fun because it makes for less manual repetition of measurements that are verified by automation and more exploration around the manual test.

In addition, the best-written manual tests are significantly different from ideal checks. Manual (or quasi-manual) tests tend to focus on scenarios or mini-scenarios, because that is the natural usage for end-users, and it gives testers the most opportunities to find issues and characterize them as bugs to be considered by the team. Checks focus on specific verifications, and ideally are as short as possible.

“Check” means that that the verifications are strictly limited to what is specified in advance, either by the coded-in verifications, verifications for the test group (if that is implemented) or by the test harness as a whole. In the context of automated testing, this specification might be specified in prose, but is always specified in the code that is written. This works very well for an automated test, because it is important to be completely consistent over test runs with what is and is not verified about the SUT.

 

This post is based on an excerpt from Matt Griscom’s forthcoming book, MetaAutomation.