Put actionable information in the automated test failure

This post follows up on yesterday’s post http://metaautomation.blogspot.com/2011/09/fail-soon-fail-fast.htm.

Metaautomation is about maximizing the value of automation to the business.

When automation fails, you don’t want to have to do the test again with a debugger or other diagnostic tool, for at least these reasons:

·         It’s time consuming i.e. expensive

·         It might demand expertise an individual tester might not have, so would pull in another person

·         On reproducing the failure, there’s a good chance that behavior will be different, maybe even in a way that isn’t obvious, so now you’ve complicated the issue and/or lost information

·         Automated analysis (a Metaautomation topic to come later) wouldn’t be useful in this hypothetical case because it’s already established that important information is missing

There will always be some un-anticipated or unprecedented failures that fail the tests and require detailed follow-up. The detailed follow-up becomes affordable if most failures don’t require a detailed, time-consuming follow-up. So, let me optimistically say that I think if the correct information is included with the failure, 90% of failures won’t require a manual repro or a debugging session.

How to get there?

Logging helps, but it’s not sufficient. Important context information for the failure can be logged successfully. If the steps executed in pursuit of use case execution are logged, and except for the last one before the failure, they represent the happy path for the automated test, they could be useful. They also must be constructed well (timestamp, and at least one level of context for execution) to be useful. Consistency in labels and structures is very important for readability and for automated failure analysis.

Exceptions are key, because done well, they provide a single object to the test code or framework that contains all of the information you care to throw into it. They’re thrown by the framework at failure time, or a custom exception is created and thrown at the time a failure/exceptional condition is encountered. Custom exceptions are typically written in product code, and can be written for test code as well.

Using the same language or framework for the test automation as you use for the product is important, because if there’s a change, information will be lost across the boundary and/or you don’t have the same power in handling, wrapping and throwing exceptions. Java will work for this, but I prefer .Net Framework with C# (or any of the other .Net language implementations e.g. VB.Net which compile to the same IL anyway).

Think about a failure condition being measured (or just encountered e.g. in the case of a network exception) and rolling up the stack. At some point, a stack trace, exception type and message will be reported and recorded. Maybe that’s enough, but information could be lost too, for example if there’s some state that pertains to the failure that just got lost as it fell off the stack. In that case, you could improve failure reporting by:

1.       Catching the exception. Catch (System.Exception) might work here, depending on the needs of the calling code …

2.       Create a string with context information for the error, in the context of the scope of the try block corresponding to the catch of step 1

3.       Create an instance of a custom exception type, where the type communicates the current context/location e.g. CompanyAbcModuleDefTestException

4.       For the constructor of the custom exception, pass the exception of step 1 (to become the innerexception of the custom exception) and the string of step 2

5.       Throw the custom exception instance

6.       Repeat steps 1-5 in as many different contexts as helps nail down the root cause of the failure, hence the action item to respond to the test failure

If this sounds like a lot of overhead, remember what it’s like to struggle to reproduce an error at 1AM just prior to code freeze for ship… and also this is a very nice kind of functional test documentation, to benefit all the different testers who might need to maintain the code for other reasons.

Now, when the test code does final reporting of the failure, all that information is available in an object to be serialized in e.g. XML or HTML to create a highly stable stream, or even objects that lend themselves very nicely both to human readability and (for the future) automated test failure analysis. Clarity into the test code base or harness helps at refactor or extension time, too.