Quick summary of MetaAutomation, the Origin and the Importance

This post is a brief explanation of the origin and importance of MetaAutomation.

Please see earlier posts for a dependency diagram of the patterns of MetaAutomation.

MetaAutomation began as an effort to stop the wasteful and expensive practice, common to conventional automation for quality, of dropping important and actionable bits on the floor. The waste, delay, and impact on team trust and communication happen every time an automated test fails and the root cause is not immediately clear.

There are three common but unproductive practices to fix at the same time:

1.       The exclusive search for bugs, at the expense of the larger quality picture, provides simple measures of productivity but neglects the end-user experience and the larger picture of quality measurement.

2.       The common practice of creating automation from manual test cases tends to cause slow, brittle automation and a failure to prioritize quality measurements by business behavior.

3.       The relevance and/or meaning of running conventional automation must be manually interpreted and communicated to the larger team, whether it passes or fails. If this communication does not happen effectively, the larger team loses trust in what the automation is doing for product quality.

MetaAutomation addresses bottom-up and end-to-end automation to measure product quality. It is a pattern language of five patterns that describe language-independent and platform-independent approaches to fix these problems and create fast, scalable, focused verifications of prioritized business behaviors, without the randomization of false negatives or false positives, with compact and robust artifacts for analysis, and actionable results directed to the people who need to know. It is a pattern language, rather than a simple set of patterns, because the five patterns have a dependency order that suggests in turn an order of implementation.

The first and least-dependent pattern of MetaAutomation is called Atomic Check. The name “check” is important because the verifications are explicitly planned and limited, rather than being open-ended as with a manual test. “Atomic” indicates that the procedure is as small and simple as possible, while still verifying the targeted business behavior and being independent of all other atomic checks.

The next pattern, Precondition Pool, manages preconditions, states, or other entities that are needed for fast and scalable check runs. This depends on another requirement of Atomic Check, that is, that the obsolete setup and teardown patterns in automated tests are eliminated. The check itself contains preliminary steps if they are needed in line with the body of the check, but if they can be managed outside of the check timing and process, they happen as part of the Precondition Pool implementation. For embedded systems, a common task for Precondition Pool is to maintain users and user settings. Doing this as part of a process that is independent of the check run allows the checks to run significantly faster.

The Parallel Run pattern depends on the checks being small, so they can be run quickly, and independent of all other such checks. Parallel Run addresses the means to scale the checks across any number of independently-running environments, so that they can run simultaneously and scale with resources.

Smart Retry addresses the many one-off failures that occur with automated tests, especially with external resources or race conditions external to the system under test (SUT). The precise and detailed artifacts from a run of an atomic check allow Smart Retry to decide, based on configuration of the automation system for the SUT, whether a check is automatically re-tried, and on completion, whether a given check result has been duplicated. This approach eliminates check failures that are not actionable, i.e. randomization of team members’ time, and improves the quality of, and trust in, the data that automation is creating to define the quality of the SUT.

Automated Triage depends on the precise and detailed data that the Atomic Check pattern requires of the checks, and with a system that includes data on the scope of the SUT and team members’ areas of responsibility. This pattern represents an important part of the “meta” of MetaAutomation: communications with action items from running automation on the SUT can be automatically directed, in effect, themselves automated. This increases the precision and detail of communications around the team on current quality issues, while eliminating some of the manual work that would otherwise be required.

Some aspects of what Automated Triage can do for the team are unnecessary however, if the new approach to quality regression that MetaAutomation offers enables the automated tests to run so much faster that they can gate check-ins to the team code repository. Gated check-ins prevent regressions in quality from impacting the larger team.

In case the techniques of MetaAutomation are applied to mainline code that is visible to the whole team, it automatically characterizes and reproduces quality regressions and therefore greatly accelerates the fixes to these issues, which again minimizes the impacts of breakages on the larger team.

Common failure patterns of automation efforts can now be avoided. Automation of bottom-up and end-to-end tests can now be part of the success story of ensuring that the quality of the SUT is always getting better; what works correctly for the SUT is well-understood and well-communicated. Finding bugs is still important, but now bugs take their appropriate place in the overall quality assessment of the SUT.

Performance data is now neatly persisted as well; if recorded, it is now stored as part of the compact, strongly-typed and structured artifacts, as part of running the usual automation. No special testing is needed, and more data on performance is available for analysis and over a longer part of the development cycle for the SUT.

Manual testing is accelerated, made more productive and less boring, because manual testers now have access to detailed data on verifications for core business behaviors, meaning that the manual testers never have go to steps to verify these details and are instead free to do more exploratory testing and find more bugs.

The power, simplicity and flexibility of MetaAutomation make it into an inevitable part of modern quality measurement for impactful products. For more information, the book on MetaAutomation is available on Amazon.