Innovative MetaAutomation

Is MetaAutomation actually innovative?

From the Wikipedia entry on “Innovation:”

Innovation is a new idea, more effective device or process. Innovation can be viewed as the application of better solutions that meet new requirements, inarticulated needs, or existing market needs. This is accomplished through more effective products, processes, services, technologies, or ideas that are readily available to markets, governments and society. The term innovation can be defined as something original and more effective and, as a consequence, new, that "breaks into" the market or society.

By this definition, yes, MetaAutomation is very innovative. Here are some ways that MetaAutomation speaks to the Wikipedia entry:

New Idea

What happens after or as a result of an automation run is now explicitly important, with details on why this is true and what can be achieved with strong, focused data on the automation run. It goes far beyond what can be achieved with conventional automation or even automation that leaves no more than a flight-recorder log and a Boolean result as artifacts.

You could even call that the “Meta” of MetaAutomation.

MetaAutomation includes several paradigm shifts:

1.       Manual testing and automated testing bring different value to the team, and if done effectively, they will excel in different ways. If automation is focused on what it does well, it has quality value far beyond what the team gets by automating manual test cases.

2.       For automation artifacts, a strongly-typed, focused, pure data solution is much better than a flight-recorder log stream: more compact, richer information, presentation can be applied later so it is flexible (this is especially easy with XML and XSL, and optionally XSD as well, all of them W3C standards), and much more powerful in downstream solutions e.g. analysis, flexible reporting and automated communications.

Ever tried to automate parsing of streamed flight-recorder logs, where the format of the log entries includes English grammar and there are few format rules? I have. It’s nearly impossible to do this reliably.

Even given established practices, current investments, and the dissonance of introducing ideas that are very new and different, MetaAutomation (or, something a lot like it) is inevitable for software that matters to people.

More Effective Process

MetaAutomation includes the process or processes by which quality information is communicated and even made actionable and interactive around the team. This really only works when automation is reliable, fast and scalable, clear and actionable.

Without MetaAutomation, there tends to be failed automation that is ignored because it is assumed, usually correctly, that automation failure is not about a product quality issue. If the automation has recently failed, resolution to an action item often must wait until one of the automation authors has time to reproduce the problem and step through to debug the issue, which again is usually nothing to do with actual product quality.

Inarticulated Needs

To my knowledge, the need to have automation be reliable, actionable, self-documenting and interactive throughout the team has not ever been articulated as part of one concept (although, the reliable part has been attempted before at least, and the interactive part exists, depending on team process). Quality information that covers all business-logic behaviors and persists data for efficient and robust analysis over any time span represented by the data? That might exist for specific proprietary applications such as aviation or military, but it’s not publicly available, and in any case, I suspect that MetaAutomation does it better.

My last blog post addressed the connection between MetaAutomation and Sarbanes-Oxley (SOX) compliance, and in turn, company valuation (for a publicly-traded software company). This is a real value, but AFAIK it has not been articulated before.

Readily Available

MetaAutomation is a pattern language, a series of five design patterns with a defined dependency. It’s language-independent, although the freely-available sample implementation is written in C#, as is the library and sample that is nearly complete. The upcoming lib puts performance information into the XML elements that describe the steps of the check, and is distributed so can run a single check against multiple (virtual) machines.

The book defines the patterns with UML diagrams, clear reasoning, references etc. and describes the sample code.

Original

The closest analog I can find to MetaAutomation is the book “xUnit Design Patterns” by Meszaros. This book is useful, and I reference it from the MetaAutomation book, but there are major differences as well:

xUnit Design Patterns is a survey of existing practices. It does a nice job of describing the patterns and in some cases connecting them together, but the patterns are not original with Meszaros. It’s a large set of patterns, most of which I would not recommend following for specific engineering and quality reasons I won’t go into here.

Some of the patterns of MetaAutomation (e.g. Precondition Pool, Parallel Run) are related to existing patterns and practices in software engineering. Smart Retry is related to a Retry pattern in test automation practice, but MetaAutomation is able to do it much more efficiently because it depends on Atomic Check, the least dependent and possibly most original pattern of MetaAutomation.

The pattern language is very original, and it’s necessary to show the dependency. With Atomic Check, you can achieve a strong return in the value of your quality measurements, but also, if you choose, you and your team can go on to implement the other patterns of MetaAutomation.

Breaks

MetaAutomation is not incremental, nor additive. It’s not a tool you can buy.

I’m not saying that existing automation practices don’t add value; in most cases, they do.

I am saying that the value of MetaAutomation can only be achieved by approaching automation differently from conventional practices. The adoption bar is high because MetaAutomation might require you to start over with automation for software quality.

Innovation

For me, MetaAutomation has been a labor of love. I love software that works, and I love addressing big-picture issues with intensity, tenacity and intellectual courage. It hasn’t been easy, and sometimes my personality pushes people away, but I’m proud to think differently.

 

MetaAutomation and Sarbanes-Oxley (SOX)

SOX is about accuracy and transparency in accounting and protection for investors in publicly-traded companies. The standards of SOX were enacted as US federal law in 2002.

Remember the collapse of Enron? SOX prevents that kind of thing.

It was such a great idea that it was subsequently imitated in many other countries around the world.

SOX Title III is on corporate responsibility, including the accuracy and validity of corporate financial reports. Section 302 (in Title III) mandates a set of “internal controls” which in turn have requirements for timeliness, accuracy and completeness of internal communications at a company about assets and operations.

SOX Title IV is on financial disclosures, and requires internal controls assuring accuracy and completeness. Section 404 focuses on risk assessment and disclosure of the effectiveness of a company’s internal controls.

At a software company, or a company that creates software as part of the business, these controls are part of the company’s information technology controls or IT controls.

MetaAutomation creates very strong stories for risk management through:

1.       Complete, detailed and accurate assessments of software product quality, focused on business requirements of the system

2.       Actionable quality events around regressions, found and delivered fast enough to prevent or quickly fix failures found by automated testing

3.       A very detailed, searchable and presentable record of software quality that uniformly spans time and all the business behaviors of the product that are accessible to automated testing

The “…timeliness, accuracy and completeness of internal communications…” on quality issues of software development assets is assured with MetaAutomation, to a greater degree than possible with any kind of automation that only creates English-grammar flight-recorder logs. For developing software, on the quality side, Section 302 is covered!

For “…risk assessment and disclosure…” same thing. Visibility and interactivity with the quality data is very high. Section 404 is covered, too!

MetaAutomation reduces the cost of SOX compliance while improving corporate governance. Research has shown that this has a significant positive effect on company valuation (see This paper, and if you don't have access, it's easy and free to sign up.)

Quote from the paper: “The overall regression results are consistent with the view that SOX has a favourable long-term favourable impact.”

The adoption costs of MetaAutomation are not trivial, but improved company valuation is potentially quite significant.