Automation is key, but don't automate manual tests!

A common pattern in software quality these days is to consider automation as if it were the equivalent of manual testing. With this view, the question of “what to automate” is easy; the team proceeds to automate the existing test cases. These were written as manual test cases, and probably run occasionally too or maybe just as part of developing the test cases. It’s thought that the manual test cases don’t have to be a burden any more on the manual testers, because once automated, the quality measurements that is the subject of those test cases is now taken care of. But, is it safe to assume this?

 

An important difference is that testers notice stuff; they’re smart and observant. Automated tests, on the other hand, only “notice” what is explicitly coded into the automated test, that is, other than the application crashing or some confusing implicit verifications e.g. a null reference exception.

 

Manual tests of a GUI or a web page can be automated, but the value of running that test as automation is very different than running it manually. The automated test might be faster, but it misses all sorts of details that would be very obvious to a human tester, and is prone to a brittleness that that manual test does not suffer. An experienced team would run the “automated” manual test periodically anyway, to confirm all those qualities that the automation probably doesn’t cover (or, if it did cover, would become too brittle to be useful).

 

It doesn’t make sense to automate manual tests. But, quick regression is important to manage the risk around product churn. So, what to automate?

 

A simple but deterministic approach is to verify elements of business behavior, and only the intermediate steps as needed. This way, the automation is faster and more robust and is clear about what it verifies and what it does not.

 

This is the first of several important requirements of the Atomic Check pattern, the least dependent pattern of MetaAutomation. There are other wonderful properties of Atomic Check, but this one is summarized in an activity diagram on page 37 of the book on MetaAutomation (and shown below).