Monday, October 17, 2011
Automate Business Logic First
At PNSQC 2011 last week, I met some very interesting and smart characters. One of them was Douglas Hoffman, current president of the Association for Software Testing. (some links: http://www.softwarequalitymethods.com/,
One of Douglas’ observations is that API-level E2E testing is 10 times as fast as graphical user-interface (GUI) level testing. He knows this from a very impressive array of industry experience.
Alan Page writes: “For 95% of all software applications, automating the GUI is a waste of time.” http://blogs.msdn.com//b/alanpa/archive/2008/09/18/gui-schmooey.aspx
I agree. For a software product in production or a project under development, assuming that there is some degree of separation between the GUI and the business logic it depends on, it’s quicker and more effective to automation the business logic, not the GUI. Some would call this automating the application programming interface (API).
I currently have the privilege of working on a team that does this right: the GUI layer is thin in terms of logic. The business happens below the API.
Here are some things that make automation that includes the GUI expensive:
· GUI design can change often, because it’s what is displayed to end users. GUI is complex and laden with personal values. Whenever there’s a change in the GUI, any automation that depends on it must be fixed.
· GUIs must be localized, and this usually means much more than just changing the displayed strings, introducing an additional level of instability to GUI automation.
· GUI automation is rife with race conditions due to the nature of display.
· Brian Marick: “Graphical user interfaces are notoriously unstable.” http://www.stickyminds.com/getfile.asp?ot=XML&id=2010&fn=XDD2010filelistfilename1%2Edoc
· Automating the GUI takes many more dependencies on the product than automating the API, because the GUI is much more complex than the API. A result of this is that the GUI automation is much less atomic than API automation (see an earlier post http://metaautomation.blogspot.com/2011/09/atomic-tests.html), therefore riskier.
· GUI automation requires an entire display mechanism, at least in software, even if the window is hidden. This involves significant overhead.
I’ve never seen stable GUI automation. What I’ve seen instead is that it’s expensive to keep going with GUI automation, if the team cares about keeping it running.
I’ve seen this many times: automation which includes the GUI fails, and it’s understood – usually correctly, but not always – that it’s just a GUI problem causing instability. This can have the effect of hiding deeper product problems, which can introduce a lot of risk to a project.
Here are some reasons to automate the business logic instead:
· API automation is simpler and more transparent to write
· API automation is much more stable
· API automation runs 10 times as fast as GUI automation (from Douglas Hoffman again) (although, in my experience the difference is even larger)
· There is no display overhead
· With API automation, the compiler is your friend (assuming you’re using a strongly-typed language, which you should be. See this post http://metaautomation.blogspot.com/2011/10/patterns-and-antipatterns-in-automation_03.html)
· Failures in API automation are much more likely to be actionable and important …
This last point is huge: If API automation fails for some reason other than for dependency failures, timeouts or race conditions (e.g. as mentioned here http://metaautomation.blogspot.com/2011/09/intro-to-metaautomation.html ) then it’s due to some change in business layers of the product and this instantly becomes a quality issue. It might be due to some intentional design change that the developers are making without first telling Test, but just as often it’s a significant quality issue that is worth filing a bug – and in that case, the team just found a quality failure essentially instantly, so it can be fixed very quickly and downstream risk is minimized. If it’s your automation that found the bug, you’re a hero!
Here’s another reason to focus on the business logic:
I’ve heard it said that it’s better to automation the GUI, because then you’re automating the whole product. At a shallow level, this has the ring of truth, but consider this perspective instead: suppose your team focuses on automating business logic, and therefore gets much more quality information, quicker regression, better coverage etc. Then, a GUI bug is found, and this bug is found later in the software development life cycle (SDLC) than otherwise, but no worries: the risk of deferring a GUI bug fix is very low, because if the application is at all well-designed, none of the business logic depends on the GUI flow.
Manual test will never go away, and people are smart and able to spot all sorts of GUI issues that automation can’t do without huge investment in automated GUI checks. Therefore, the GUI bugs are likely to be found by manual testers anyway, and they’re still relatively low risk because the rest of the product doesn’t depend on the GUI.
This is why I’m happy to focus on business logic automation.