MetaAutomation starts with making automation failures actionable, maximizing the value of automation results, and continues by automating triage. MetaAutomation reduces the cost of fixing existing automation and ensures that automation helps your quality measurements and improvements, rather than hindering them.
Tuesday, May 12, 2015
From the Wikipedia
entry on “Innovation:”
Innovation is a new
idea, more effective device or process. Innovation can be viewed as the
application of better solutions that meet new requirements, inarticulated needs,
or existing market needs. This is accomplished through more effective products,
processes, services, technologies, or ideas that are readily available to
markets, governments and society. The term innovation can be defined as
something original and more effective and, as a consequence, new, that
"breaks into" the market or society.
By this definition,
yes, MetaAutomation is very innovative. Here are some ways that MetaAutomation
speaks to the Wikipedia entry:
What happens after or as a result of an automation run is
now explicitly important, with details on why this is true and what can be
achieved with strong, focused data on the automation run. It goes far beyond
what can be achieved with conventional automation or even automation that
leaves no more than a flight-recorder log and a Boolean result as artifacts.
You could even call that the “Meta” of MetaAutomation.
MetaAutomation includes several paradigm shifts:
1.Manual testing and automated testing bring
different value to the team, and if done effectively, they will excel in
different ways. If automation is focused on what it does well, it has quality value
far beyond what the team gets by automating manual test cases.
2.For automation artifacts, a strongly-typed,
focused, pure data solution is much better than a flight-recorder log stream:
more compact, richer information, presentation can be applied later so it is
flexible (this is especially easy with XML and XSL, and optionally XSD as well,
all of them W3C standards), and much more powerful in downstream solutions e.g.
analysis, flexible reporting and automated communications.
Ever tried to automate parsing of streamed flight-recorder
logs, where the format of the log entries includes English grammar and there
are few format rules? I have. It’s nearly impossible to do this reliably.
Even given established practices, current investments, and
the dissonance of introducing ideas that are very new and different,
MetaAutomation (or, something a lot like it) is inevitable for software that
matters to people.
More Effective Process
MetaAutomation includes the process or processes by which
quality information is communicated and even made actionable and interactive
around the team. This really only works when automation is reliable, fast and
scalable, clear and actionable.
Without MetaAutomation, there tends to be failed automation
that is ignored because it is assumed, usually correctly, that automation
failure is not about a product quality issue. If the automation has recently
failed, resolution to an action item often must wait until one of the
automation authors has time to reproduce the problem and step through to debug
the issue, which again is usually nothing to do with actual product quality.
To my knowledge, the need to have automation be reliable,
actionable, self-documenting and interactive throughout the team has not ever
been articulated as part of one concept (although, the reliable part has been
attempted before at least, and the interactive part exists, depending on team process).
Quality information that covers all business-logic behaviors and persists data
for efficient and robust analysis over any time span represented by the data?
That might exist for specific proprietary applications such as aviation or
military, but it’s not publicly available, and in any case, I suspect that
MetaAutomation does it better.
My last blog post addressed the connection between
MetaAutomation and Sarbanes-Oxley (SOX) compliance, and in turn, company
valuation (for a publicly-traded software company). This is a real value, but
AFAIK it has not been articulated before.
MetaAutomation is a pattern language, a series of five
design patterns with a defined dependency. It’s language-independent, although the
freely-available sample implementation is written in C#, as is the library and
sample that is nearly complete. The upcoming lib puts performance information
into the XML elements that describe the steps of the check, and is distributed
so can run a single check against multiple (virtual) machines.
The book defines the patterns with UML diagrams, clear
reasoning, references etc. and describes the sample code.
The closest analog I can find to MetaAutomation is the book “xUnit
Design Patterns” by Meszaros. This book is useful, and I reference it from the
MetaAutomation book, but there are major differences as well:
xUnit Design Patterns is a survey of existing practices. It
does a nice job of describing the patterns and in some cases connecting them
together, but the patterns are not original with Meszaros. It’s a large set of
patterns, most of which I would not recommend following for specific engineering
and quality reasons I won’t go into here.
Some of the patterns of MetaAutomation (e.g. Precondition
Pool, Parallel Run) are related to existing patterns and practices in software
engineering. Smart Retry is related to a Retry pattern in test automation
practice, but MetaAutomation is able to do it much more efficiently because it
depends on Atomic Check, the least dependent and possibly most original pattern
The pattern language is very original, and it’s necessary to
show the dependency. With Atomic Check, you can achieve a strong return in the
value of your quality measurements, but also, if you choose, you and your team
can go on to implement the other patterns of MetaAutomation.
MetaAutomation is not incremental, nor additive. It’s not a
tool you can buy.
I’m not saying that existing automation practices don’t add
value; in most cases, they do.
I am saying that the value of MetaAutomation can only be
achieved by approaching automation differently from conventional practices. The
adoption bar is high because MetaAutomation might require you to start over
with automation for software quality.
For me, MetaAutomation has been a labor of love. I love
software that works, and I love addressing big-picture issues with intensity,
tenacity and intellectual courage. It hasn’t been easy, and sometimes my
personality pushes people away, but I’m proud to think differently.