Linear Check Steps are Good, but Hierarchical Steps are The Way of the Future

A check (what some call an “automated test”) has steps. Typically, these steps are linear: first step 1, then step 2, then etc. That’s useful, because now you know how the check does its thing, assuming you know what the words describing the check steps mean.

If your software development team values communication, maybe they’ve defined a Ubiquitous Language (UL) of terms that are meaningful to the problem space that the product addresses and meaningful to all team members.

The steps of the check document what it does, so define simple steps for the check using terms of your UL. If the steps are too granular for your UL terms, then lump the linear steps together until your steps are expressed with your UL.

Now the check steps use UL, and that’s good, but what happens inside each of those steps? If you lumped steps together to make a linear list of steps, each of which has a meaning in your UL, the steps themselves aren’t trivial. What goes on inside the step? It’s documented in code somewhere, but not generally available to the team. What if the step changes subtly, so that the steps work together in a sequence for different test cases? This happens; been there, done that.

Hierarchical check steps avoid these problems, and greatly improve communication around the team. With MetaAutomation, the check steps are self-documenting in the original hierarchy with results “pass,” “fail,” or “blocked.” The customer of the check results does not always want to drill down to the child steps, but (s)he always has that option, and it would be quick and easy if needed.

I’ll make a more concrete example of the value of hierarchical steps and drill-down capability:

I live in north Seattle. Suppose I want to visit my brother near Boston. Suppose further I’m afraid of flying, and I want to drive a car.

I go to Bing maps, type in the addresses, and presto! Driving directions.

But, there are 40 linear steps to get there, and I presume that they’re all correct, but most of them are not even close to being in the domain language of driving a car for that distance. For example, step 8 is “Take ramp right and follow signs for I-5 South.” If someone asked me “How do I drive to Boston?” and I replied something including “Take ramp right and follow signs for I-5 South” the person would make excuses and go ask someone else.

No, the domain language for driving to my brother’s house includes terms like this:

1.       Do you know how to get to I-5? OK, then

2.       Take I-5 South to I-90

3.       Take I-90 East to eastern Massachusetts

4.       I-95 North to Melrose

5.       …(several more steps)

There are 7 linear steps total. But, there’s a problem with step 3: Are we really taking the shortest, most optimized route? If yes, take I-94 through North Dakota. If doing the more obvious, take I-90 through South Dakota. Which is it? This could be important if the car breaks down and help is needed.

If the driving steps are hierarchical, the customer of that information (the person consuming the directions) has choices: the default view of the steps shows the 7 top-level steps, the same steps as if they were just linear. But, all the information about driving to Boston is there, not just the broad steps.

With hierarchical information, the customer can drill down at step 3 to discover that the most optimized route means getting off I-90 at Billings Montana to take I-94 instead through Custer, Bighorn, etc. and stay on I-94 all the way east to Tomah Wisconsin. If one were driving the route, or if you got the call that the driver is lost or broken down, that would be important information.

Back to software quality:

If a check fails, having the hierarchical information could make the difference between the need to reproduce the problem (if that’s even possible, which in some cases it’s not) and debug through, or having all the information one needs to enter a bug, fix the problem, or write it off as an external transient timing issue (race condition). The hierarchical information makes Smart Retry (a pattern of MetaAutomation) possible because it will make it clear whether a problem is reproduced or unique, external or owned by the developer team.

Hierarchical check steps gives the team the flexibility to use UL terms whenever possible so the whole team benefits from knowing what’s going on with the product, and at the same time have child steps that are finely granular to reflect exactly how the product is being driven, or exactly what the value of a parameter is, or what the result is from some instrumentation etc.

Hierarchical check steps also record performance data for all of the steps, so if a step is running slow, the customer can always drill down to discover which child step is running long.

Performance data in line with the most granular steps is recorded every time those steps are run, so analysis over time can uncover performance limits and trends.

As software gets ever more important in our lives, and more complex at the same time, a better solution for recording and communicating quality is inevitable.

Edit October 27th, 2015:
See also this post on hierarchical check artifacts
http://metaautomation.blogspot.com/2015/10/automated-verifications-deserve-better.html