Patterns and antipatterns in automation (part 1 of 2)

Patterns are common ways of solving frequent problems in software.

From the introduction to the famous “Gang of Four” patterns book “Design Patterns” (Gamma, Helm, Johnson, Vlissides): Design Patterns make it easier to reuse successful designs and architectures.

In Test, patterns are used to follow well-established best practices. Antipatterns are patterns of action “ … that may be commonly used but is ineffective and/or counterproductive in practice” http://en.wikipedia.org/wiki/Anti-pattern.

For example, one pattern of action is to automate as many test cases as quickly possible because that’s the performance metric that management is using, and we all want to look productive for the business. When the test cases start failing, ignore them and continue automating test cases, because that’s the metric that management is using.

This is actually an antipattern because it’s counterproductive practice. The business value of automating the product (or parts of the product) is to exercise and regress behavior (make sure it’s still working per design) for parts of the product on a regular schedule. If that can’t be done (because the automation broke) the value of those cases goes away or even worse: the quality team proceeds as if quality was being measured for that part of the product, when it’s not because those relatively high-priority test cases are failing.

There’s a closely-associated antipattern which I’ve addressed in practice for years but for which I credit my friend Adam Yuret (his blog is here http://contextdrivenagility.com/) for help in crystallizing: test cases are mostly written as if they were manual test cases, and they have value for running through the test case as a human tester and observing the results with human perception and intelligence. If this manual test case is automated, the verifications (aka “checks”) are typically many fewer than what a human might perceive, and they might even be zero i.e. nothing is getting verified and the value of running that automated test is also zero.

Adam maintains that what was a (manual) test case is no longer a test case because the entire flow of the user experience (UX) is no longer being measured; there are zero or more checks that are being measured, but they must explicitly be written, coded, and tested by the person doing the test automation. By default, they’re not.

I disagree with the “test case is no longer a test case” judgment, but this does point out an important truth about quality practice: the value of manual testing never goes away (unless maybe if the product has no graphical user-interface aka GUI).

The antipattern here shows up in two common and related team management practices: the idea that automating a GUI test case means that that a certain part of the product never has to be visited manually ever again, or that the practice of manual testing from the GUI (and, the manual testers) can simply be automated away.

I’ll continue this post tomorrow with more patterns vs. antipatterns…