Tuesday, October 7, 2014

Moving forward with automation technology, using the word "Check"

Bach and Bolton (their blog post, “Testing and Checking Refined”) describe the idea that the word “check” is a useful label for some types of activities and procedures commonly called “test.” The issue is that when measuring product quality, it is important to differentiate between what test professionals typically do as they exercise the product, and the value of automated measurements of product behavior with Boolean pass/fail results.

In literature about testing, the word “check” often serves as synonym for “assert” or “verify.” This book proposes a closely related use of the word, similar to what Bach and Bolton described, and an important one to the nature of the first pattern of MetaAutomation, Atomic Check.


Many practitioners believe that automation for software quality starts with manual test cases. The manual test case is designed to be executed by a person, and often they are, with useful quality measurement results.

When the manual test case is automated, the conventional wisdom goes, the quality measurement value of that manual test is multiplied many times, because the test can now run faster and more reliably and offline or after hours. In practice, “faster” is true most often, but not always, and “more reliably” is only sometimes true, but running the tests offline and repeatedly is in any case a very significant business value for catching regressions and managing product risk.

On execution by a person, the original manual test had some quality value to the product team. Is that value now covered by the automated version, so the manual test never needs to be run again? No, generally, this is not the case; people are smart and observant, and test professionals can note and characterize (write a bug for) quality issues that automation will not notice, especially for apps with a GUI or web sites. On execution by a test professional, and according to expectations of people performing that role, the original manual test measures much more than just the written test procedure and verifications. People notice stuff, especially good testers. Automation only notices what is required for the automation to run, plus explicitly coded verifications or assertions.

If the team automates a manual test, and thereafter no person ever runs the original test, along with the well-understood gains of automation comes a significant but poorly understood loss in team capability of measuring quality. The person who might have taken time to run the manual test now has more time to add value in other ways, but all of the human-observable aspects of product quality that are implied by or incidental to the manual test are now going unmeasured. The automation can add verifications, but speaking from extensive experience in automating for quality, coded verifications of product quality that do not break the automation are very limited in number relative to the range of issues a tester can notice.

Automating manual tests, combined with the above misunderstanding, creates significant quality risk.

An example

Here is a manual test for a hypothetical team working on a banking web site. Call these steps “transfer balances:”

1.       Browse to the bank site

2.       Login as test user

3.       Note checking balance

4.       Note savings balance

5.       Go to transfer page

6.       Transfer $5 from savings into checking

7.       Verify correct checking balance

8.       Verify correct savings balance

9.       Logout

When a person runs this test, in addition to the explicit steps and verifications, she might notice and bug a huge number of potential issues, for example:

·         The browser shows a security warning about the site certificate

·         The balance in savings has gone negative

·         The ad on the page messes up page rendering

·         An incorrect name is shown for the greeting

·         The protocol serving the page does not have SSL

·         There is a spurious alert message


Later, she automates this test, with the verifications as written. Management is happy and she is happy because, they think, she never has to run the manual test again.

It is true that the importance of running this manual test is reduced when it is automated, because what the team decided are the most important business issues are now verified automatically. However, the need for a team member to do these or closely related procedures manually does not actually go away.

The common misunderstanding is that after the manual test is automated, there is no need to run it manually or even test around that scenario. Call this case “A.” On the other hand, we have case “B,” the understanding that there must still be some manual testing around this, because the automation is very limited in ability to measure the details of a quality experience for the product.

The judgments represented by cases A and B cause different actions from the team around measuring quality, after the “transfer balances” steps from the example test have been automated. The actions and inactions in case A cause a gap in quality measurement, because the bulleted items may no longer be measured. The longer these issues are broken, the more potential for downstream issues, and for quality regressions, difficulty and cost in finding and correcting the root cause of the issue increase rapidly with the time lapse between the issue occurring in product code and discovery by the team.

That all adds up to create the risk of case A, relative to case B. Case A can potentially result in such issues as:

·         Basic product issues discovered late in the product cycle

·         Issues shipped to customers, so the customers find serious issues before the team knows about them

From a project perspective, an important root cause of the management error of case A is that “transfer balances” starts as a manual test, and when it is automated, it becomes an “automated test.” It still looks like the same test to teams afflicted by the misunderstanding of case A, except that it now runs repeatedly and with lower personnel cost. The word “test” still applies, and that trips the team up.


Words are labels, and choice of labels is easily dismissed as “just semantics,” but words have connotations as well as denotations.

A manual test is a test. If that test is automated, the result is still a type of test, but to highlight the change that automation brings, this book uses and recommends the noun “check” for that purpose.

Checks do not have nearly the powers of observation that a person does. Any verifications that the check does can be implicit, that is, comes with the procedure code anyway, or explicit, which requires an explicitly coded verification.

Use the term “check” every time a measurement is made of the product where

1.       It is an end to end test

2.       The measurement procedure runs without human presence or intervention

3.       The measurement procedure completes with a pass/fail result

The word “check” works as a noun, for example: “A check is a better label to use than automated test.” It also works as a verb, for example: “Execute this set of automation to check that the end-to-end product behavior is still correct.”

This terminology clarifies what automated testing does, but even better, it avoids eclipsing manual testing around the functionality.

Promoting the term “check” instead of “automated test” emphasizes the limitation of automation, and makes it clear that, especially when working with a web site or other GUI, some manual testing still needs to be done. Checks make the manual testing easier and less tedious by removing the need to check the important business-logic behaviors of the product, but they do not remove the need to run the manual tests or to do exploratory testing around the tested feature.

“Check” is still a kind of test, but think of it like cheese that originated in the Brie region of France; one can call it “cheese” and be correct, but the much preferred and more efficient term is “Brie” and people sound more discerning, educated and domain-aware when they use the latter term.

Readers may be wondering at this point: How about unit tests? Should we call them “unit checks” now?

This book uses “unit test.” Unlike with automated end-to-end tests, that were originally written as manual tests but then automated, unit tests are never manual in origin so the risk described above is not an issue.

This book also uses “check” for an API or service test, as long as all dependencies are in place. This is useful for techniques such as bottom-up testing, for which the Atomic Check pattern is especially powerful.

Another advantage of “check” is that it makes it easier to see that the best checks are designed and grouped differently than manual tests. There is much more on this point in the book “MetaAutomation,” in Chapter 3 on the pattern Atomic Check.



  1. Matt,

    I'll disagree a bit with your point about "unit test". Sure they may not have been written down beforehand, but they still fall within the realm of a "check" because they are validation points (assertions) within the code. These types of automated tests just exist at a different level of the system (code level) and are "quantitative" in nature because they just validate/verify something (did I get the right result, or did I hit this section of code).

    I guess I'm olde school. I tend to be one of the "clumpers" of terminology and don't like splitting hairs too much. Especially with people outside of the testing world, it just confuses them when we split hairs on the discussion.

    As you said, a "check" is a type of "test". It is a validation/verification type of test, or as some in the natural sciences would call a "quantitative" factor. You validate/verify some known (or unknown) quantity (how much exists). And as you said, automation of the execution of said "tests" does give us a quantitative analysis of the reliability of the SUT.

    Whereas Exploratory methods look for a "qualitative" factor of what does or does not exist. Qualitative analysis leads to determining "what" exists or not. Which is good, and this does require a human brain to make the determination after scientific experimentation and investigation. A computer doesn't have that same cognitive ability like the human brain. We'd have to wait for full AI to get that (hopefully it won't be SkyNet).

    I think the least of our concerns right now is defining the differences between "checks" and "tests". When it comes to automation we need to focus on the bigger issues of misconceptions and false expectations by other groups. After that then we can start to narrow some of the vocabulary and meaning.


    Jim Hazen

  2. Thanks for your thoughtful note, Jim.

    On "unit test:" Yes, I can see it both ways. Maybe "unit check" is a better description, but OTOH and as you also note, defining differences might not be important or certainly not the most important thing. People have been saying "unit test" for a long time, and IMO the term still works fine and there isn't sufficient reason to advocate for changing it.

    IMO human testers are not capable of doing just a check, and I think there's not sufficient reason to complicate their task (which is important and quite challenging to do well, anyway) so I do not like using "check" to describe something that people do, other than Boolean operations ("Hey, would you check that the radio is turned on?").

    I agree with you on "... we need to focus on the bigger issues of misconceptions and false expectations by other groups." and unless I misunderstood you, the label change from "automated test" (for the E2E case) to "check" helps enough to justify the (optional, and limited) change.

    "Check" is for an automated procedure where there's no intelligent testing going on (no AI, and no people). For purposes of that statement, all people are smart, and they would all notice stuff and not sit there and starve in case of difficulty (as B&B wrote).

  3. Matt,

    I'm of the opinion that if folks want to call it "automated checking", or "automated verification" like I like to call it, its still a test.

    At first I was also almost convinced that the change mattered. But I have since thought about it much more and based on this reflection no longer believe that the change in terminology is relevant nor desired by testers at large. I applaud you including it in your book and hey, if it takes off, you'll be famous (or infamous) :)

    All kidding aside, when it comes to automation I'm with Jim in that we need to focus on instructing what the reality is to the stake holders. Rather than wasting time on vocabulary. If testing is a young field then automated testing is even younger; the way I see it we're still trying to see what works and what doesn't. We're in prototype stage when it comes to automation now and what it will be in the future.

  4. Yup, automation for software quality is a kind of testing, and yup, it's semantics.

    The misunderstanding I wrote about is a real risk, though. I've had a lot of experience and frustration with that risk. It's significant.

    Saying "check" instead of "test" is a tool to advance the state of the art. It's a bit like saying "climate change" rather than "global warming." Climate change is a better label for what's going on, because it represents reality with less chance of misunderstanding.

    It's like calling Saturn a "planet" rather than a "star." Or, "footprint" rather than "environmental impact from human activity."

    As you write "...we need to focus on instructing what the reality is to the stake holders." Agree. I think "automated verification" works too, but "check" is pithier. :)

  5. It was very nice article and it is very useful to Manual Testing learners.We also provide Cub training software online training.


Note: Only a member of this blog may post a comment.