The first think you might be wondering is: what’s metaautomation?
Automation of course is getting the software product or system under test (SUT) to do stuff without a person driving the SUT directly, e.g. while the team sleeps or parties, and with zero or more checks of product behavior with Boolean results that get reported somehow. I’m thinking of E2E or scenario tests, not e.g. unit, perf or stress tests. A failure in basic automation is often useful only as an indicator that the test team has more work to do.
Metaautomation is taking it to the next level: basically making automation failures actionable, up to and including automated triage.
The second thing you might be wondering: why is this important to me, the SDET and advocate for product quality?
In a typical software creation scenario there are two prominent metrics in use by the test team: number of test cases and number of test cases automated. The test automaters are under some pressure to deliver as many automated tests as they can, but in a week or day, some or most automated steps begin to fail. The report that shows the flipped bits – the automated test cases that are now failing – but doesn’t give much more info than that, so at some point the test automater has to stop automating new test cases and go back and try to fix the failures. This can take as long as the original automation for that test effort, so sometimes it isn’t done at all – the failed automation continues failing as before, and has little, no or even negative value to any useful measure of product quality.
When you’re the owner and you get word of an automation failure, a variety of questions might come to mind:
Is the root cause a product failure, an automation failure, a dependency failure, a race condition beyond team ownership, or something even more complex?
Has this root cause failure happened before?
Would it happen again? Is it even reproducible, reliably or at all?
If it looks like a product failure, does it require debugging the product, or just the test code, to get more information before you pass the issue on as a bug?
If root cause is or might be beyond team control (but, it’s not an issue for the product end-user) will you just try again to see if it passes the 2nd time?
Metaautomation addresses all of these issues. Note that, unlike automation, metaautomation depends on knowledge of product implementation, e.g. the source code.
You could think of it like this: the relationship of metaautomation to automation is analogous to the relationship of automation to manual test. Manual test is one of the first things you do to measure product quality, and the importance of manual test never goes away, but the power of the team in maintaining and promoting product quality is significantly enhanced with good product automation (including at least minimal reporting of the automation results).
Manual test is costly in terms of people and time, and it’s never really reliable because people are human.
Automation addresses the time cost (by running automation) the boredom (by having a computer, lab, or cloud processing, do the work for you) and some other issues as well (e.g. frequency of the measurement of the product, consistency, reliability…). That’s why automation is so big these days. Everybody in software knows the value, so everybody’s doin’ it.
Unfortunately, when automation fails, manual analysis can be very costly in terms of time. It might mean loading the test code, and the product, hooking up or simulating external dependencies and doing a debugging session. A complex or difficult bug can take hours or days of follow-up. Sometimes, in consideration of these costs, it’s not even done at all.
Metaautomation addresses these costs and, if done well, can increase team efficiency and make product quality stronger.
If you’re doing good automation, you’re probably already doing the most basic kind of metaautomation: logging, as in having your test code and/or product stream, at least, hard-coded messages to a text-based file. This is a kind of metaautomation because it’s orthogonal to having the product do stuff and test runtime verifications; it’s the first piece of post-run test analysis, and it can be used to feed other pieces of metaautomation that I’ll be getting into with future posts, although there are better sources of information (which will be addressed in future posts as well).
Metaautomation isn’t the answer to all scenarios. There will still be some manual analysis, just like there will still be some manual testing. But, metaautomation speeds analysis by reducing the need to examine the product or test code (especially by saving the need to debug through the code) and it can speed triage of issues as a direct result (depending on team process). Automated automated (yep, 2nd-order automated) test re-runs and automated test failure analysis are other pieces of metaautomation that can speed human-based triage further, or even bypass it.
The third thing you’re probably wondering is: why do we need a new word for this? Are you kidding me - six @#! syllables?
Automation is summarized nicely here:
Note that the article briefly discusses reporting, but not analysis. Most of the value of automation comes after the automation is created: that’s where metaautomation can help a team shine!
The plan for this blog:
I plan to post every day or so for as long as it takes to (re)define
· Quality
· Test
· Metaautomation, the what and the how (this will take roughly a dozen posts)
· Values in automation
· … I have other ideas, and the blog should have momentum by this time …
Following posts will tend to be shorter and might be less dense.
Oh, yeah, note too:
This blog is not associated in any way with the industrial and marine company http://www.meta-automation.com/ although, I do like boats, especially sea kayaks.
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.