Wednesday, October 28, 2015

Analytics and MetaAutomation

Analytics are very powerful and an awesome developing technology, and your team needs automated verifications of business requirements as well (e.g. using the Atomic Check pattern of MetaAutomation, and dependent patterns).

Why both?

Atomic checks are fast and reliable enough to verify requirements from an end-user perspective quickly, and a set of them are even fast and scalable enough to gate check-ins (especially with the Precondition Pool, Parallel Run and Smart Retry patterns of MetaAutomation). Implementing these patterns can ensure that quality is always moving forward, which is especially important for software that matters.

Analytics can give the team ideas of what the end uses are experiencing. This is awesome, because otherwise you’d have little or no idea of what’s going on for your end users. This is very valuable for, e.g., A/B testing. OTOH, it’s somewhat like the shadows of Plato’s cave: you have to reconstruct and guess what’s really going on in some cases, e.g., through session abandonments on a site.

Analytics need product instrumentation, aka telemetry. Good news; there’s synergy here! Automated verifications can use those as well. Analytics benefits from product instrumentation through logs, and automated verifications can use the same outputs but synchronized for inclusion in the artifact of a check run. The valid XML document created by an atomic check is ideal for this.


This diagram gives a view of how product instrumentation can benefit both analytics and automated verifications (checks).

Tuesday, October 27, 2015

Automated Verifications Deserve Better than Flight-Recorder Logs

This post is a follow-up from this one:

"Automated Test" is an oxymoron, because you're not automating what a manual tester could do with the script, and unlike true automation (industrial, operations or IT automation) you don't care about the product output (unless you're doing your quality measurement that way explicitly).

Automation for software quality has some value at authoring time because it's a good way of finding bugs, but really it only has business value with the verifications being done, most of which are coded explicitly (preliminary verifications, and a target verification or verification cluster, per MetaAutomation).

So, call it what it is: automated verifications! That understanding opens up tremendous business value for software quality! If done right, it can help manual testers too by making their jobs more interesting and more relevant.

According to tradition, and according to the origins of "test automation" from industrial automation, the output of automation the software system under test (SUT) is flight-recorder logs and, maybe, a stack trace. But, there's no structure there; it's just a stream of information, some relevant, but most not relevant or interesting.

It's time to leave that tradition behind! Automated verifications have well-defined beginnings and ends. They're bounded in time! Use that fact to make some structure out of the artifacts, i.e., the information that flows from an automated verification.

XML is ideal for this, especially with an XSD schema. With an XML document, there are boundaries and a hierarchical structure, and if the presentation is left out (as it should be) then it's pure data; it's compact and strongly-typed, available for robust and efficient analysis, and completely flexible for presenting the data.

Even better: The least-dependent pattern of MetaAutomation, Atomic Check, shows how to write self-documenting hierarchical steps for your checks (i.e., automated verifications) which means that you'll never have to choose between using your teams domain-specific language (DSL) or ubiquitous language, and using those (sometimes, very important) granular details of what the check is doing with your product or where/why it failed. You can do both!

With self-documenting hierarchical steps, after a given check has succeeded all the way through at least once, if the check fails, all the check steps are recorded with "pass," "fail," or "blocked." This gives another view of course on what the check was trying to do and tells you exactly what parts of product quality are blocked or not measured due to the failure.

Self-documenting hierarchical steps also give you information that's detailed and precise enough to know whether the specific failure has happened before! See the Smart Retry pattern of MetaAutomation for how to do and use this.

Even more: using a structure like XML for your check artifact means that you have a ready placeholder for perf data; it's recorded how many milliseconds a given step took, for every step in the hierarchy. Perf testing doesn't need a separate infrastructure or focus, and perf data is available for robust analysis over the SDLC.

See the samples on to see how this works.


Tuesday, October 20, 2015

Linear Check Steps are Good, but Hierarchical Steps are The Way of the Future

A check (what some call an “automated test”) has steps. Typically, these steps are linear: first step 1, then step 2, then etc. That’s useful, because now you know how the check does its thing, assuming you know what the words describing the check steps mean.

If your software development team values communication, maybe they’ve defined a Ubiquitous Language (UL) of terms that are meaningful to the problem space that the product addresses and meaningful to all team members.

The steps of the check document what it does, so define simple steps for the check using terms of your UL. If the steps are too granular for your UL terms, then lump the linear steps together until your steps are expressed with your UL.

Now the check steps use UL, and that’s good, but what happens inside each of those steps? If you lumped steps together to make a linear list of steps, each of which has a meaning in your UL, the steps themselves aren’t trivial. What goes on inside the step? It’s documented in code somewhere, but not generally available to the team. What if the step changes subtly, so that the steps work together in a sequence for different test cases? This happens; been there, done that.

Hierarchical check steps avoid these problems, and greatly improve communication around the team. With MetaAutomation, the check steps are self-documenting in the original hierarchy with results “pass,” “fail,” or “blocked.” The customer of the check results does not always want to drill down to the child steps, but (s)he always has that option, and it would be quick and easy if needed.

I’ll make a more concrete example of the value of hierarchical steps and drill-down capability:

I live in north Seattle. Suppose I want to visit my brother near Boston. Suppose further I’m afraid of flying, and I want to drive a car.

I go to Bing maps, type in the addresses, and presto! Driving directions.

But, there are 40 linear steps to get there, and I presume that they’re all correct, but most of them are not even close to being in the domain language of driving a car for that distance. For example, step 8 is “Take ramp right and follow signs for I-5 South.” If someone asked me “How do I drive to Boston?” and I replied something including “Take ramp right and follow signs for I-5 South” the person would make excuses and go ask someone else.

No, the domain language for driving to my brother’s house includes terms like this:

1.       Do you know how to get to I-5? OK, then

2.       Take I-5 South to I-90

3.       Take I-90 East to eastern Massachusetts

4.       I-95 North to Melrose

5.       …(several more steps)

There are 7 linear steps total. But, there’s a problem with step 3: Are we really taking the shortest, most optimized route? If yes, take I-94 through North Dakota. If doing the more obvious, take I-90 through South Dakota. Which is it? This could be important if the car breaks down and help is needed.

If the driving steps are hierarchical, the customer of that information (the person consuming the directions) has choices: the default view of the steps shows the 7 top-level steps, the same steps as if they were just linear. But, all the information about driving to Boston is there, not just the broad steps.

With hierarchical information, the customer can drill down at step 3 to discover that the most optimized route means getting off I-90 at Billings Montana to take I-94 instead through Custer, Bighorn, etc. and stay on I-94 all the way east to Tomah Wisconsin. If one were driving the route, or if you got the call that the driver is lost or broken down, that would be important information.

Back to software quality:

If a check fails, having the hierarchical information could make the difference between the need to reproduce the problem (if that’s even possible, which in some cases it’s not) and debug through, or having all the information one needs to enter a bug, fix the problem, or write it off as an external transient timing issue (race condition). The hierarchical information makes Smart Retry (a pattern of MetaAutomation) possible because it will make it clear whether a problem is reproduced or unique, external or owned by the developer team.

Hierarchical check steps gives the team the flexibility to use UL terms whenever possible so the whole team benefits from knowing what’s going on with the product, and at the same time have child steps that are finely granular to reflect exactly how the product is being driven, or exactly what the value of a parameter is, or what the result is from some instrumentation etc.

Hierarchical check steps also record performance data for all of the steps, so if a step is running slow, the customer can always drill down to discover which child step is running long.

Performance data in line with the most granular steps is recorded every time those steps are run, so analysis over time can uncover performance limits and trends.

As software gets ever more important in our lives, and more complex at the same time, a better solution for recording and communicating quality is inevitable.

Edit October 27th, 2015:
See also this post on hierarchical check artifacts

Friday, October 9, 2015

What is MetaAutomation? A Very Brief Technical Overview

What, again, is MetaAutomation all about?

Here is a metaphorical view of the potential of MetaAutomation.

This post is about a technical view.

The least dependent pattern of MetaAutomation is called Atomic Check. There are three main aspects that Atomic Check specifies and they’re somewhat bound together:

First, the check must be atomic i.e. as short and as simple as possible while still measuring the targeted business requirement for the software product. The result of a check run is a Boolean of course, pass or fail.

Second, no check has a dependency on any other check. They can be run in any order and on any number of clients, to scale with available resources.

Third, every check run creates a detailed result, which I call an artifact. Whether it’s pass or fail, the artifact has all of the following characteristics:

1.       It’s pure data. (I use valid XML with an XSD schema.)

2.       All the individual steps of the check are self-documented in the artifact.

3.       Check steps are hierarchical, for better clarity, stability, and robustness in analysis.

4.       Check steps, and the check as a whole, can have additional data in name/value pairs.

5.       Each check step element in the XML has perf data: the number of milliseconds that step took to complete (including child steps, if there are any).

The third aspect is the most radical. What happened to your flight-recorder log? It can be there if you want the help at development time, but that’s the only setting where it’s useful. Afterward, it’s discarded, because the pure-data artifact is more compact, more precise and more accurate, and much better for subsequent analysis and other cool stuff.

Why would I ask you to do something radically different? Because it helps you do very cool and powerful things, like communicate to all roles in the team in as much or as little detail as they want! It enables analysis of product behavior over the entire SDLC, greatly simplifies and normalizes perf information, makes for better information on software asset value for Sarbanes-Oxley (SOX) …

The third aspect I describe above, in combination with the first two, enable the dependent patterns of MetaAutomation to work and deliver quality value to the whole team that’s an order of magnitude better than conventional practice. For software that matters to people, MetaAutomation or the equivalent practice is inevitable.

Please see here for more information on MetaAutomation.

What is MetaAutomation? A Brief Metaphorical View

What is MetaAutomation all about?

MetaAutomation opens a conceptual door, and through that door, it shows how to create powerful quality and communications value for a software team, faster and better, more complete, robust, reliable and satisfying.

That door is currently being held closed by several factors at least:

Conventional wisdom and practice maintains that “automated testing” is just like “manual testing,” in fact, according to this meme the words “automated” and “manual” can be dropped from the software quality profession. There are good intentions here… but, it comes with business risk, and it’s pulling the door closed.

There are tools vendors and open-source projects which support “automating” tests which are originally designed as manual tests, i.e., for a person to execute to observe quality information on a software product under development. There are good intentions here too, but it’s also risky, and it’s pulling the door closed.

The “Context-Driven School” view of testing discourages software quality professionals from seeing general approaches to software quality. This pulls the door closed.

Keyword-driven testing is represented by many tools and techniques, BDD, Cucumber and Gherkin for example, but requires linear runtime interpretation of an invented language (keywords or phrases) to run. This limits both the precision and the accuracy of the artifacts of a check run, which in turn limits value of the automated verifications for communication and for quality measurement. The concept is very popular these days – it does have value! – but it still looks a lot like automating manual tests, with all the risks and limitations that go along with that, and the artifacts aren’t much better. This movement, too, pulls the door closed.

Open the door! Automated verifications is in many ways a distinct discipline with unique powers, relative to other software quality patterns and practices. For repeatable regression checks, for performance data, for managing business risk and shipping software faster and with better quality, for quality communications around all team roles and across distributed teams, there’s huge value on the other side of that door.

That’s what MetaAutomation is all about.
Please see here
for a very brief technical overview.
 (edited 10.12.2015)

Thursday, October 8, 2015

Automated Verifications are Special, and Why This Is Important

MetaAutomation enables automated verifications to be much more powerful than they are with existing practices. However, to realize that value requires, first, an important paradigm shift.

Conventional wisdom about automation, according to current practices, is that there is no fine distinction that separates automated verifications from other aspects of software quality, e.g. manual testing or human-led testing that uses tools (in addition to the SUT, which is itself a kind of tool).

This view can simplify software quality planning because it approaches “test automation” as if this were natural, rather than the contradiction-in-terms that it really is, and that automation is therefore simply an extension of the manual testing effort.

The simplification is an attractive model for understanding the very complex and difficult problem space around software quality. One could argue that this even follows the philosophical principal of Occam’s razor, i.e., the simpler model that explains the observations is the one more likely to be correct or useful.

However, fortunately or unfortunately (depending on your perspective), understanding automation as an extension of the manual testing effort does not explain experiences or capabilities in the space. People with experience in the various roles of doing software quality know well that:

People are very good at

·               Finding bugs

·               Working around issues and exploring

·               Perceiving and judging quality

·               Finding and charactering bugs

But, they’re poor at

·               Quickly and reliably repeating steps many times

·               Making precise or accurate measurements

·               Keeping track of or recording details

Computers driving the product (system under test, or SUT) are very good at

·               Quickly and reliably repeating steps many times

·               Keeping track of and recording details

·               Making precise and accurate measurements

But, computers are poor at

·               Perceiving and judging quality

·               Working around issues or exploring

·               Finding or characterizing bugs

There’s a big divergence between what people are good at, and what computers are good at. This is just one set of reasons that “test automation” doesn’t work; most of what people do can’t be automated. Besides, the term “automation” that comes from industrial automation (building things, faster and more accurately and with fewer people), automation in an airplane (handling and presenting information, and assisting the pilots in flying the plane), or automation in an operations department at a company (allocating network resources through scripts), does not apply to software quality; automation is about output, and if a software product is under development, we generally don’t care about the product output, aside from quality measurements. The “automation” focus of making or doing stuff does not apply to software quality generally.

There is one thing that can be automated, however: verifications around behavior of the software product. Given the above lists on what computers can do well, if we program a computer to do a quality measurement, we’re limited to what eventually amounts to a Boolean quantity: pass or fail.

Note that this is different from using a tool or tools to measure the product so that a person makes the quality decision. People are smart (see above) and they have observational and emotional powers that computers do not. That’s not “automated testing,” either, because all you’re doing is applying a tool (in addition to the software product itself) to make a quality decision. Using a tool could be “automation” like the true meaning of the word (i.e. producing things, modifying or presenting information), but by itself, application of that tool has nothing to do with quality. What the person does with the application of the tool might be related to quality, though, depending on the person’s role and purpose.

I describe the new paradigm like this:

1.       Characterizing software product quality is a vast and open-ended pursuit.

2.       The observational, emotional and adaptive powers of people is indispensable to software quality. (I call this “manual test,” for lack of a better term, to emphasize that it’s a person making quality decisions from product behavior.)

3.       The only part of “automation” that honestly applies to software quality is automated verifications.

4.       Manual test, and automated verifications, are powerful and important in very different ways.

5.       Recognizing the truth of #4 above opens the team up to vast increases in quality measurement efficiency, communication, reporting and analysis.

My last point (#5) is the reason I invented MetaAutomation.

Suppose the quality assurance (QA) team has a Rodney Dangerfield problem (no respect!). MetaAutomation can get them the respect they deserve, by improving speed, quality, transparency of what the software product is doing, exactly what is being measured and what is not being measured. Their achievements will be visible across the whole software team, and the whole team will be grateful.

Suppose the accounting department is preparing for a Sarbanes-Oxley (SOX) audit of the company. They want to know about the value of software developed (or, under development) in the company: what works? How reliably does it work? How fast is it? MetaAutomation answers that need, too, with unprecedented precision and accuracy.

But, MetaAutomation requires step #4 above; it requires people to recognize that automated verifications for quality are very different from manual (or, human-decided) testing.

Once you accept that automated verifications have a special difference and a distinct potential, you open your team to new horizons in productivity and value.

Thursday, September 24, 2015

Manual Test to MetaAutomation, Continous Testing and DevOps

A software test is some procedure with intent to measure quality.

When the test procedure is documented, usually the expected result is a pass or fail, but it could be some other basic measurement as well e.g. time for completion in the case of a simple performance test.

With exploratory testing, the result could be any issue that the person doing the exploration perceives as an issue, usually limited to issues that may be actionable by the team at some point in time.

There are many other kinds of test as well, and many people who occasionally fulfill the testing role by being open to recording, for the benefit of the team and the product, some issue of quality. Anybody with responsibility for the product, aka system under test or SUT, can do this role.

There are many tests where the procedure is automated and the pass/fail criterion are both automated, i.e. “scripted.” These can be fast and cheap to run, but the only information recorded is the result (the “artifact”) of the test. There are many issues that humans could potentially observe while they are testing the product, but for such issues which are difficult, risky, or expensive to code for detection, or just aren’t anticipated by whoever is scripting the test, recording doesn’t take place.

For example, while testing a REST service, a human tester might notice that performance is usually quite good but periodically slows down significantly. This can be worth recording as a “bug” for investigation and possible correction by the tester. But, a scripted test of that same service might not record any such pattern, other than a well-set time limit which fails the test prematurely sometimes so records a subset of the information that a human would perceive. A human could always decide “well, let me time this” and get out a stopwatch app to record the timing of the service response, but no scripted test could do this.

For another example, suppose a tester is working on a credit union portal, and notices that on the login page, the page layout is not acceptable in some way. A scripted test is very unlikely to include a measurement that would perceive this, and for good reason; such a script would be complex, slow and flaky.

Such tests are often classified as either “manual” or “scripted.” (There is also the term “automated” but that term is problematic, as I’ll describe below.) With the manual tests, testers can perceive, characterize and record a wide range of issues. This is true whether or not a stopwatch app or some other tool is used (in addition to the SUT, which of course is also a kind of tool); the human tester perceives issues. People are smart that way.

With the scripted tests, the quality issues that are noted are limited to what is explicitly coded as a measurement (aside from “the SUT crashed” which fortunately is less common these days). The number of measurements are best limited for many reasons, e.g.

·         If a quality measurement fails, but there are other quality measurements that are never made, those other quality measurements get hidden and not measured

·         The script must be fast

·         The script must be simple and robust as possible


Scripts written from manual tests tend to be brittle when up against complexities that human testers barely need to think about, e.g. race conditions or design or implementation changes.

The term “check” describes a type of test that is entirely scripted, including the quality measurement that is the target verification of the check. The term “check” is useful for many reasons, but is very important for avoiding a business risk that results from a misunderstanding:

Many tests are written as manual tests i.e. the original intent, and usually the original “run” or execution of the test, is by a person on an interface that is designed for humans e.g. a GUI or a web page or site. A very common practice is to take these tests and “automate” them i.e. write script that makes the SUT do just as if a human were running the tests, or as close as possible. Management and the team then assumes that the quality risk that would be covered by manual testers running the manual tests, are now covered by the “automation,” and they understand that there is no need to run the tests manually any more … but, they are mistaken.

Using the word “check” avoids this misunderstanding, because it’s clear (at least, more clear) that the only quality measurements happening are the ones explicitly coded into the newly-scripted procedure. This is similar to the distinction between “manual” and “scripted” or “automated” testing: “manual” testing means that it’s a person who perceives issues or decides how the test result is reported. People are smart! Computers and scripted procedures: not smart! (This leaves artificial intelligence aside for the moment, because it doesn’t apply here, nor for the near future.)

This is a good thing for clarity: manual testing will always be important for what that role is good at (e.g. finding bugs), and scripted testing is good for what it’s good at (regressing quality and making sure it all still works); actually, it can be an extremely powerful driver of quality, as I explain below.

Here is another good reason to not use the word “test” when we’re really talking about scripted verifications: the common phrases “test automation,” “automated test,” and “automated testing” are all oxymorons.

The term “automation” comes from industrial automation. Automation is all about producing a product or products faster, more accurately, or more cheaply, and with fewer people than before. This is used in IT departments as well, with no physical product like a factory would have, but in that case as well, the end product is the goal of the automation.

“Test automation” is about making the SUT do stuff, but nobody cares about the product output (unless that is needed for a quality measurement); it’s in development. Actually, people do care about quality measurements of the product related to what the product does, but that often has nothing to do with the by-design results of the product. If data is used to drive the product, if it looks like real-world data, it’s usually faked or anonymized for business and quality reasons. The word “automation” is a poor fit for software quality, so disassociate it from the modifying word “test.”

“Check” is a more accurate, and less risky, term to use. “Scripted verification” works too, and emphasizes that mostly what one is doing with scripted checks is simply verifications.

It might seem now that there’s a downside to checks or scripted verifications; since we know that humans notice stuff that checks do not, and that checks need to be short and simple to be useful, why write any scripting for quality measurements at all?

This answer is well-known: the scripted checks are highly repeatable, they can be fast, and they can (sometimes) be made very reliable. The results of the scripts are recorded for record-keeping and simple analysis and follow-up.

There’s another answer that builds on that, and as yet is not well-known: if the checks are written in a certain way, scalability, robustness and reliability, analysis and communication around the team can be vastly more effective.

The foundational, least dependent (and the most innovative) pattern of MetaAutomation is “Atomic Check.”

“Check” simply means that it’s a scripted (or, automated, if you will) verification.

“Atomic” means that it’s as simple as possible while still achievement measurement of the targeted behavior of the SUT.

Atomic Check describes exactly how to design the optimal check for each of your prioritized business requirement. It has these benefits as well:

·         Checks are all independent of each other, for scaling

·         Check artifacts are pure data, e.g. XML, providing for efficient and effect analysis across the product and SDLC and customized presentations for different roles across the team

·         Check steps, and their results (pass, fail, blocked) are self-documenting

·         The artifacts from check failures are highly actionable, so minimize the need to reproduce errors, including transient errors

·         They are generally written in the same language as the SUT, so there is no need to learn (or invent) a new language

DevOps is powerful in what it can do for shipping quality software faster. Continuous testing is the test/QA side of DevOps. MetaAutomation is the optimal way to do continuous testing.

Oh, and remember how important the manual testing role is? Atomic Check self-documents exactly what is working, so relieves manual testers of repetitive drudgery, allowing them to focus on what they’re good at. Duplicative work goes away.

More information on the MetaAutomation pattern language is here:

and, in other posts on this blog.

Thursday, August 6, 2015

Stronger Quality with MetaAutomation: Part 3 of 3, Quality through the SDLC

Good, efficient communication is an important asset to the team. MetaAutomation shows how to achieve this over the short and long term.

Imagine a test/QA team that creates a quality data store for the whole team with these properties:

·         Compact records

·         Validated data structures

·         Performance data in-line with check steps and results

·         Check steps marked pass, fail, or blocked

For example, from an intranet site, any team member can query and do analysis directly on the data store. The pure, structured data shows exactly what checks were run and when, and how long each executed step took in milliseconds, as well as the check as a whole, even if the check itself was distributed for multi-tier steps or verifications in the check.

This level of quality detail starts when the system is up and running, and extends throughout the SDLC or through all iterations, depending on the process the team uses to develop and ship their product. Since the check steps are self-documenting, the steps laid out in the check artifact are as stable as the code that runs the check.

With focused, pure and structured data on product quality, including all of the self-documenting steps of a given check, it’s clearly known what’s working and what the verifications are. Trust and communication are greatly improved, between geographically-distributed teams and between the test/QA team, developers, program managers, and leadership.

The quality of the company’s software assets, i.e., the product under development, is clearly expressed in great detail. This helps with SOX compliance as well, so the managers and investors are happy.

The test/QA team gets the exposure and respect it deserves.

Successes and failures in the business requirements of the product are archived in great detail over time, so improvement in quality in the product is evident. Product owners and leaders have a powerful asset to help them manage risk, and they know that failures in core behaviors of the product are fixed quickly, both in theory and in practice. It’s all there in the quality record.

By addressing the business value of programmatically-driven verifications of software product quality, for developing software that matters, MetaAutomation radically increases the value of the quality role to the business. Open-source software will be available this summer to demonstrate scalable, distributed checks and show how to do the same for your team.

Wednesday, August 5, 2015

Stronger Quality with MetaAutomation: Part 2 of 3, Handling a Check Failure

What happens on your team when a check (what some call “automated test”) fails?

If you follow the common practice of automating manual test cases, then the authors of the automation are expected to follow up and diagnose the problem. Usually, it’s a failure in the automation itself, everybody on the team knows that, so it doesn’t get much priority or respect generally, but even if it does, it’s very time consuming and labor-intensive to resolve the failure.

Alternatively, the author of the automation watches the automation proceed to see if something goes wrong. That shortens the communication chain, but it’s very time-consuming, expensive, and doesn’t scale at all.

Regression tests or checks that are effective towards managing quality risk must be capable of sending action items outside the test/QA team quickly. False positives, i.e., messages on quality issues that ultimately turn out to not concern the product at all, are wasteful and they corrode trust in the test/QA team. Therefore, quality communications must be quick and trustworthy for test/QA to be effective.

On check failure, people can look at flight-recorder logs that lead up to a point of failure, but logs tend to be uneven in quality, verbose, and not workable for automated parsing. A person has to study them for them to have any value, so the onus is on test/QA again to follow up. Bottom-up testing, or testing at the service or API layer, helps, but the problem of uneven log quality remains. Mixing presentation with the data, e.g., English grammar or HTML, bloats the logs.

Imagine, instead, an artifact of pure structured data, dense and succinct, whether the check passes or not. Steps are self-documenting in a hierarchy that reflects the code, whether they pass, fail, or are blocked by an earlier failure.

MetaAutomation puts all of this information in efficient, pure data with a schema, even if the check needs to be run across multiple machines or application layers.

A failed check can be retried immediately, and on failure, the second result compared in detail to the first. Transient failures are avoided, and persistent failures are reproduced. Automated analysis can determine whether the failure is internal or external to the project, and even find a responsible developer in the product or test role as needed.

If so configured, a product dev would receive an email if a) the exact failure was reproduced, and b) the check step, stack trace, and any other data added by check code indicates ownership.

Atomic Check shows how to run end-to-end regression tests so fast and reliably, they can be run as check-in gates in large numbers. Check failures are so detailed, the probability that a problem needs to be reproduced is small.

With MetaAutomation, communications outside test/QA are both quick and trustworthy. See parts 1 and 3 of this series for more information.

Tuesday, August 4, 2015

Stronger Quality with MetaAutomation: Part 1 of 3, Fast Quality

Manual testing and programmatically-driven quality measurements are both very important for software quality, however, they are each good at very different things.

Please notice that I’m avoiding the word “automation” here; there’s good reason for that. I write more on this topic here

Freedom from the word “automation,” and liberation from assumptions that programmatically-driving quality measurements should be like manual testing in any way, have similar benefits: They open up new frontiers in productivity for managing quality and risk for your software project.

Imagine focusing on prioritized business requirements, at the software layer closest (if at all possible) to where those business items are implemented. Writing just one check – that is, a programmed verification – per business requirement, makes for simpler, faster checks.

This is one of the reasons I created MetaAutomation: to highlight more effective techniques to programmatically measure and report on the quality of the SUT.

The least-dependent pattern of the MetaAutomation pattern language is Atomic Check. This pattern shows specific techniques to create a focused verification, or verification cluster, on an item of business logic. There are no verifications other than the target verification, i.e., the target of the check, and whatever check steps are needed to get to the target.

Simple, focused checks run faster and contain fewer points of failure. Atomic Check also describes how to make the check independent of all other checks, so that running your set of checks will scale across resources and your selected set of checks can be as fast as you need them to be.

Atomic Check also creates an artifact of the check run in strongly-typed and pure data, e.g., a grammar of XML. This has many benefits, including that it enables another very useful pattern, called Smart Retry, which I’ll write more about in part 2, and addresses visibility, quality analysis, and SOX compliance, which I’ll discuss in part 3.

Tuesday, July 14, 2015

Automation Gets No Respect

The world of software automation for quality is riddled with failures. When the people creating the automation of the software under test (SUT) fail to create reliably-running tests, or it becomes clear that this effort takes more time than first estimated, management and the rest of the team lose confidence. In any case, driving the SUT through scenarios is too often seen as a risky, low-value afterthought. After all, the developers can test the product themselves and learn things that they thought the test team was going to tell them anyway, but for some reason can’t reliably deliver.

In any case, the conventional approach to software automation for quality creates a losing situation for the people doing the work.

If they are told that the highest-value automation is end-to-end automation of the product, including the web page or GUI, they are likely doomed to write a system that creates many false positives – i.e., test failures that have nothing to do with product quality - which in turn create more work for them because they must follow up with a debug session just to discover if there is an actionable piece of information for the rest of the team.

The broader team pays little attention to the results from the checks because they know

1.       False positives are common, and if there really was a product bug, the authors of the check would discover that in a debug session and tell them.

2.       The checks don’t measure what they’re designed to do, because they can’t possible match the perception and smarts of a human testing the SUT directly.

With the correct focus on verifying and regressing the business requirements of the SUT, rather than on automating the SUT do make it do stuff, the false-positive problem and the what-is-the-check-verifying problem go away. I created MetaAutomation to describe how to take the optimal approach to solving these problems and creating many other benefits along the way:

·         The focus is on prioritized business requirements, not manual tests or scenarios

·         Checks run faster and scale better with resources

·         Check steps are detailed and self-documenting, with pass, fail or blocked status recorded in the results

·         Check artifacts are pure data, to enable robust analysis on results and across check runs

·         The quality measurement results are transparent and available for sophisticated queries and flexible presentations across the whole team

·         Performance data is recorded in line with the check step results

With MetaAutomation, the test and QA role can produce the speedy, comprehensive, detailed, and transparent quality information to ensure that functional quality always gets better.

If you want to give the test/QA role the importance and respect it deserves, try MetaAutomation!

Monday, July 13, 2015

The Word “Automation” Has Led Us Astray

If you’ve written automation for software quality, there’s a good chance you did it wrong. Don’t feel bad; we’ve all been doing it wrong. It’s not our fault.
We were led astray by the word “automation.”
Automation is about automatically driving human-accessible tools to accomplish specific tasks.
When people starting programmatically driving their software system under test (SUT) for quality purposes, overloading the word “automation” seemed like the best way to describe the new practice, but actually the word applies poorly to software quality. “Automating” your SUT means making it do stuff, and that’s usually how management measures output. Quality verifications are added as an afterthought, and have little to do with the “automation” objective so they tend to be poorly planned and documented.
The word “automation” for software quality distracts people from the business value of the activity: measuring quality.
The misunderstanding that automation for software quality is just doing what humans do (i.e. manual testing), but doing it faster and more often, causes business risk:
·         Unless you’re very clear and specific on what is being measured, the quality measure is incomplete and manual testers must verify it anyway.
·         Manual tests are designed by people to be run by people. They do not make ideal automated measurements because they tend to be long, complicated, and burdened with too many or too few verifications.
Automated verifications of services and APIs tend to be more effective, but this isn’t “automation” either by the definition.
At the core of the paradigm shift is an important dichotomy:
People are very good at
·         Finding bugs
·         Working around issues
·         Perceiving and judging quality
But, they’re poor at
·         Quickly and reliably repeating steps many times
·         Making accurate measurements
·         Keeping track of details
Computers driving the SUT are very good at
·         Quickly and reliably repeating steps many times
·         Keeping track of details
·         Making accurate measurements
But, computers are poor at
·         Perceiving and judging quality
·         Working around issues
·         Finding bugs
Conventional automation for software quality misses these distinctions, and therefore comes with significant opportunity costs.
To create software that matters and be effective and efficient at measuring quality, your team must move away from conventional misguided “automation” and towards a more value-oriented paradigm. To describe this value, I created MetaAutomation.
MetaAutomation is a pattern language of 5 patterns. It’s a guide to measuring and regressing software quality quickly, reliably, and scalably, with a focus on business requirements for the SUT. The “Meta” addresses the big-picture reasons for the effort, the nature of what automated actions can do, and the post-measurement action items and efficient, robust communication where “automation” by itself fails.
MetaAutomation shows how to maximize quality measurement, knowledge, communication and productivity. Imagine e.g. self-documenting hierarchical steps automatically tracked and marked with “Pass,” “Fail,” or “Blocked.” Imagine a robust solution to the flaky-test problem!
For trustworthy, prioritized, fast, scalable, clear and around-the-team presentable quality results, try MetaAutomation!
The other posts on this blog clarify the MetaAutomation pattern language, but unfortunately the topic is also much too big for a blog post, so see also the definitive book on the topic here: