MetaAutomation starts with making automation failures actionable, maximizing the value of automation results, and continues by automating triage. MetaAutomation reduces the cost of fixing existing automation and ensures that automation helps your quality measurements and improvements, rather than hindering them.
Tuesday, October 20, 2015
Linear Check Steps are Good, but Hierarchical Steps are The Way of the Future
A check (what some call an “automated test”) has steps.
Typically, these steps are linear: first step 1, then step 2, then etc. That’s
useful, because now you know how the check does its thing, assuming you know
what the words describing the check steps mean.
If your software development team values communication, maybe
they’ve defined a Ubiquitous Language (UL) of terms that are meaningful to the
problem space that the product addresses and meaningful to all team members.
The steps of the check document what it does, so define
simple steps for the check using terms of your UL. If the steps are too
granular for your UL terms, then lump the linear steps together until your
steps are expressed with your UL.
Now the check steps use UL, and that’s good, but what
happens inside each of those steps? If you lumped steps together to make a
linear list of steps, each of which has a meaning in your UL, the steps
themselves aren’t trivial. What goes on inside the step? It’s documented in
code somewhere, but not generally available to the team. What if the step
changes subtly, so that the steps work together in a sequence for different
test cases? This happens; been there, done that.
Hierarchical check steps avoid these problems, and greatly
improve communication around the team. With MetaAutomation, the check steps are
self-documenting in the original hierarchy with results “pass,” “fail,” or “blocked.”
The customer of the check results does not always want to drill down to the
child steps, but (s)he always has that option, and it would be quick and easy
I’ll make a more concrete example of the value of hierarchical
steps and drill-down capability:
I live in north Seattle. Suppose I want to visit my brother
near Boston. Suppose further I’m afraid of flying, and I want to drive a car.
I go to Bing maps, type in the addresses, and presto!
But, there are 40 linear steps to get there, and I presume
that they’re all correct, but most of them are not even close to being in the
domain language of driving a car for that distance. For example, step 8 is “Take
ramp right and follow signs for I-5 South.” If someone asked me “How do I drive
to Boston?” and I replied something including “Take ramp right and follow signs
for I-5 South” the person would make excuses and go ask someone else.
No, the domain language for driving to my brother’s house
includes terms like this:
1.Do you know how to get to I-5? OK, then
2.Take I-5 South to I-90
3.Take I-90 East to eastern Massachusetts
4.I-95 North to Melrose
5.…(several more steps)
There are 7 linear steps total. But, there’s a problem with
step 3: Are we really taking the shortest, most optimized route? If yes, take
I-94 through North Dakota. If doing the more obvious, take I-90 through South
Dakota. Which is it? This could be important if the car breaks down and help is
If the driving steps are hierarchical, the customer of that
information (the person consuming the directions) has choices: the default view
of the steps shows the 7 top-level steps, the same steps as if they were just
linear. But, all the information
about driving to Boston is there, not just the broad steps.
With hierarchical information, the customer can drill down at
step 3 to discover that the most optimized route means getting off I-90 at
Billings Montana to take I-94 instead through Custer, Bighorn, etc. and stay on
I-94 all the way east to Tomah Wisconsin. If one were driving the route, or if
you got the call that the driver is lost or broken down, that would be
Back to software quality:
If a check fails, having the hierarchical information could
make the difference between the need to reproduce the problem (if that’s even
possible, which in some cases it’s not) and debug through, or having all the
information one needs to enter a bug, fix the problem, or write it off as an
external transient timing issue (race condition). The hierarchical information
makes Smart Retry (a pattern of MetaAutomation) possible because it will make
it clear whether a problem is reproduced or unique, external or owned by the
Hierarchical check steps gives the team the flexibility to
use UL terms whenever possible so the whole team benefits from knowing what’s going
on with the product, and at the same time have child steps that are finely granular
to reflect exactly how the product is being driven, or exactly what the value
of a parameter is, or what the result is from some instrumentation etc.
Hierarchical check steps also record performance data for
all of the steps, so if a step is running slow, the customer can always drill
down to discover which child step is running long.
Performance data in line with the most granular steps is
recorded every time those steps are run, so analysis over time can uncover
performance limits and trends.
As software gets ever more important in our lives, and more
complex at the same time, a better solution for recording and communicating
quality is inevitable.