MetaAutomation starts with making automation failures actionable, maximizing the value of automation results, and continues by automating triage. MetaAutomation reduces the cost of fixing existing automation and ensures that automation helps your quality measurements and improvements, rather than hindering them.
Monday, February 9, 2015
Automation is key, but don't automate manual tests!
A common pattern in software quality these days is to
consider automation as if it were the equivalent of manual testing. With this
view, the question of “what to automate” is easy; the team proceeds to automate
the existing test cases. These were written as manual test cases, and probably
run occasionally too or maybe just as part of developing the test cases. It’s
thought that the manual test cases don’t have to be a burden any more on the
manual testers, because once automated, the quality measurements that is the
subject of those test cases is now taken care of. But, is it safe to assume
An important difference is that testers notice stuff; they’re
smart and observant. Automated tests, on the other hand, only “notice” what is
explicitly coded into the automated test, that is, other than the application
crashing or some confusing implicit verifications e.g. a null reference
Manual tests of a GUI or a web page can be automated, but
the value of running that test as automation is very different than running it
manually. The automated test might be faster, but it misses all sorts of
details that would be very obvious to a human tester, and is prone to a
brittleness that that manual test does not suffer. An experienced team would
run the “automated” manual test periodically anyway, to confirm all those
qualities that the automation probably doesn’t cover (or, if it did cover,
would become too brittle to be useful).
It doesn’t make sense to automate manual tests. But, quick
regression is important to manage the risk around product churn. So, what to
A simple but deterministic approach is to verify elements of
business behavior, and only the intermediate steps as needed. This way, the
automation is faster and more robust and is clear about what it verifies and
what it does not.
This is the first of several important requirements of the
Atomic Check pattern, the least dependent pattern of MetaAutomation. There are
other wonderful properties of Atomic Check, but this one is summarized in an
activity diagram on page 37 of the book on MetaAutomation (and shown below).