MetaAutomation starts with making automation failures actionable, maximizing the value of automation results, and continues by automating triage. MetaAutomation reduces the cost of fixing existing automation and ensures that automation helps your quality measurements and improvements, rather than hindering them.
The software business requires shipping at some point, and
your team and business probably started out with a ship date or target for the
Here’s where quality considerations become really important:
you do NOT want your customers to discover serious issues with your software
before you do. If they do, it could be very damaging to your business.
Depending on your priorities, you might consider delaying ship for adequate
testing and QA... of course, if you read and can follow my previous posts, you
won’t need to delay J
All software ships with unfixed bugs. (Show me software that
has no bugs, and I’ll show you software that hasn’t been tested enough.) You
can’t be 100% certain that end-users (or blackhats) won’t find serious issues
with your software that would cause you regret, but you can do some things to
minimize your risk:
Ensure that you hire good people for Test early in the product cycle, and give
them the opportunity to do their best work.
Have Test present at requirement and design meetings from product
start. Their role is to make sure that the product is testable, and to minimize
the many risks of building and shipping a product.
Make sure that the Test Plan is addressed completely, and
updated as needed along the way.
When development is complete, all significant bugs have been
addressed, and you’re approaching ship time, take a week or three to exercise
the product thoroughly and make sure that all product issues that might need
fixing or that impact product quality from any perspective are addressed with
bugs, and the bugs gets triaged. Probably, most or all bugs will be deferred to
a patch or service pack, but the important thing is that you have confidence
that there aren’t serious issues that might impact customers but that are
unknown to the team. Go through the test plan and make sure that all areas of
product quality have been measured, as completely as you can in a timeframe
that’s reasonable for your business.
… if after that, there are no bad surprises, it’s ship time!
Think of bugs as atoms of actionable communication that flow
around the group with their messages about product quality. They speak only of
the product, the customer, engineering and design details.
Triage is about prioritizing the bugs and assigning them to
people if as needed to ensure that the bugs keep flowing. Triage can be done on
all bugs that haven’t been triaged yet, or all bugs assigned to certain people,
or all new and active bugs that aren’t assigned to anybody.
A leader in the group or a meeting with enough knowledge of the product,
engineering issues around the product, end-users, any intermediate customers
and other context to make edits if needed to the severity (how important is the
issue?) and priority (which issues should be addressed first?). Bugs get
assigned to developers for fixing, or program managers for design
considerations, or back to testers for more information or as “won’t fix” or
postponed as needed.
Test and dev need to work more closely when Test finds blocking
bugs that prevent major scenarios from working, or regression bugs that block
scenarios that were working before.
Test is always responsible for removing bugs from the flow
by verifying and closing them. In some groups, testers are the only ones that
can create bugs, but it’s generally OK for anybody to create bugs as long as
they’re good bugs.