This post is a brief explanation of the origin and
importance of MetaAutomation.
Please see earlier posts for a dependency diagram of the
patterns of MetaAutomation.
MetaAutomation began as an effort to stop the wasteful and
expensive practice, common to conventional automation for quality, of dropping
important and actionable bits on the floor. The waste, delay, and impact on
team trust and communication happen every time an automated test fails and the
root cause is not immediately clear.
There are three common but unproductive practices to fix at
the same time:
1.
The exclusive search for bugs, at the expense of
the larger quality picture, provides simple measures of productivity but
neglects the end-user experience and the larger picture of quality measurement.
2.
The common practice of creating automation from
manual test cases tends to cause slow, brittle automation and a failure to
prioritize quality measurements by business behavior.
3.
The relevance and/or meaning of running
conventional automation must be manually interpreted and communicated to the
larger team, whether it passes or fails. If this communication does not happen effectively,
the larger team loses trust in what the automation is doing for product quality.
MetaAutomation addresses bottom-up and end-to-end automation
to measure product quality. It is a pattern language of five patterns that
describe language-independent and platform-independent approaches to fix these
problems and create fast, scalable, focused verifications of prioritized
business behaviors, without the randomization of false negatives or false
positives, with compact and robust artifacts for analysis, and actionable
results directed to the people who need to know. It is a pattern language,
rather than a simple set of patterns, because the five patterns have a
dependency order that suggests in turn an order of implementation.
The first and least-dependent pattern of MetaAutomation is
called Atomic Check. The name “check” is important because the verifications
are explicitly planned and limited, rather than being open-ended as with a
manual test. “Atomic” indicates that the procedure is as small and simple as
possible, while still verifying the targeted business behavior and being
independent of all other atomic checks.
The next pattern, Precondition Pool, manages preconditions,
states, or other entities that are needed for fast and scalable check runs.
This depends on another requirement of Atomic Check, that is, that the obsolete
setup and teardown patterns in automated tests are eliminated. The check itself
contains preliminary steps if they are needed in line with the body of the
check, but if they can be managed outside of the check timing and process, they
happen as part of the Precondition Pool implementation. For embedded systems, a
common task for Precondition Pool is to maintain users and user settings. Doing
this as part of a process that is independent of the check run allows the
checks to run significantly faster.
The Parallel Run pattern depends on the checks being small,
so they can be run quickly, and independent of all other such checks. Parallel
Run addresses the means to scale the checks across any number of
independently-running environments, so that they can run simultaneously and
scale with resources.
Smart Retry addresses the many one-off failures that occur
with automated tests, especially with external resources or race conditions
external to the system under test (SUT). The precise and detailed artifacts
from a run of an atomic check allow Smart Retry to decide, based on
configuration of the automation system for the SUT, whether a check is
automatically re-tried, and on completion, whether a given check result has
been duplicated. This approach eliminates check failures that are not
actionable, i.e. randomization of team members’ time, and improves the quality of,
and trust in, the data that automation is creating to define the quality of the
SUT.
Automated Triage depends on the precise and detailed data
that the Atomic Check pattern requires of the checks, and with a system that
includes data on the scope of the SUT and team members’ areas of responsibility.
This pattern represents an important part of the “meta” of MetaAutomation:
communications with action items from running automation on the SUT can be
automatically directed, in effect, themselves automated. This increases the
precision and detail of communications around the team on current quality
issues, while eliminating some of the manual work that would otherwise be
required.
Some aspects of what Automated Triage can do for the team
are unnecessary however, if the new approach to quality regression that
MetaAutomation offers enables the automated tests to run so much faster that
they can gate check-ins to the team code repository. Gated check-ins prevent
regressions in quality from impacting the larger team.
In case the techniques of MetaAutomation are applied to
mainline code that is visible to the whole team, it automatically characterizes
and reproduces quality regressions and therefore greatly accelerates the fixes
to these issues, which again minimizes the impacts of breakages on the larger
team.
Common failure patterns of automation efforts can now be
avoided. Automation of bottom-up and end-to-end tests can now be part of the
success story of ensuring that the quality of the SUT is always getting better;
what works correctly for the SUT is well-understood and well-communicated.
Finding bugs is still important, but now bugs take their appropriate place in
the overall quality assessment of the SUT.
Performance data is now neatly persisted as well; if
recorded, it is now stored as part of the compact, strongly-typed and
structured artifacts, as part of running the usual automation. No special
testing is needed, and more data on performance is available for analysis and
over a longer part of the development cycle for the SUT.
Manual testing is accelerated, made more productive and less
boring, because manual testers now have access to detailed data on
verifications for core business behaviors, meaning that the manual testers
never have go to steps to verify these details and are instead free to do more exploratory
testing and find more bugs.
The power, simplicity and flexibility of MetaAutomation make
it into an inevitable part of modern quality measurement for impactful
products. For more information, the book on MetaAutomation is available on
Amazon.
