Smart Retry Your Automated Tests for Quality Value

If you automate a graphical user interface (GUI) or a web browser, you’re very familiar with this problem: there are many sporadic, one-off failures in the tests. Race conditions that are tricky or impossible to synchronize and failures from factors beyond your control or ownership break your tests, and the solution too often is to run the test again and see if it passes the 2^nd time.

The result is dissonance and distraction for whoever’s running the tests: there’s another test failure. Does it matter? Do I just have to try it again? I’ll try it again, and hope the failure goes away.

Imagine transitioning your job from one where most issues that come to your attention are not actionable (e.g. “just ignore it, or try it again and hope the issue goes away”), to one where most issues that come to your attention are actionable. That sure would help your productivity, wouldn’t it?

I wrote about this topic here in some detail:

http://metaautomation.blogspot.com/2011/09/metaautomation-is-this-failure-that.html

Now’s a good time for your organization to bring it up again. Smart Retry is an aspect of 2^nd-order MetaAutomation:

http://metaautomation.blogspot.com/2012/11/metaautomation-grows-up-new-refined.html

Smart Retry is very valuable for your productivity and communication around the organization, but if you want to get there, you need two things which each have significant value in themselves:

1.       Atomic Tests http://metaautomation.blogspot.com/2011/09/atomic-tests.html and see here too http://metaautomation.blogspot.com/2013/09/how-to-find-right-size-for-your.html 

2.       Tests that fail fast with good reporting http://metaautomation.blogspot.com/2011/09/fail-soon-fail-fast.html

And:

3.       A process with some programmability to run your tests for you and make decisions based on the results

On item 3: If you are running your tests in parallel on different machines or virtual machines or in the cloud, you will have this already, and if you don’t have this, you will because the business value makes it inevitable.

For a distributed system, you will need also a non-trivial solution for this:

4.       A service that provides users for given roles from a user pool, for time-bound use with an automated test

A Smart Retry system is an automated solution to substitute for a big piece of human judgment: whether to just run the test again, vs. taking a significant action item on it. It adds a lot of business value in itself, and it also complements other systems that scale and strengthen the Quality story of your organization.