There’s a meme going around, mostly out of Google it seems,
that “Test is Dead.”
The prototypical example of this is gmail. The special SDLC (software
development lifecycle) of gmail, for purposes of this meme, goes like this:
devs write some feature. Devs do unit testing and some E2E testing, check in,
and move on. The code is deployed to live on some (not all) web servers on the
farm. End-users notice issues, end users have nothing else to do so they use the “Report
a bug” control on the page to send a bug back to Google, Google receives the
bug and the report is well-written with sufficient detail, but not too much, so
the bug can be prioritized and potentially fixed. Tada! Testing has been
outsourced to customers.
… except that the conditions that must be true for such a
scenario to work tightly limit the applicability of this technique. See for
example this link, which discusses the security implications of this approach: http://metaautomation.blogspot.com/2011/10/no-you-cant-outsource-quality-detour.html.
The end-users must know exactly what to expect from whatever product it is, and
they’re not going to read a manual or requirements spec, so the functionality
must be a reworking of some well-known idea, say, an in-browser email client or
an online game of checkers. No automation is available, so regressions might go
undetected for a while and be more expensive to fix than otherwise, and fixing
the regression might even break the feature set with code changes that causes
the regression in the first place. Clearly, this technique is much too risky
for important or mission-critical data e.g. financial or medical data.
But, there’s one idea here that does work and is worth
elaborating: devs are accountable to do some degree of E2E testing.
Imagine if dev implements a feature, sees that unit tests
pass, thinks “good enough” and checks it in. Assume that E2E tests are not
written for this feature, because hey, it’s a brand-new feature. Build of the
product in the repository succeeds. (Assume the team is not doing branch
development.) Whoever tests that build first finds some issues, and writes
bugs, and puts them on the developer’s plate. The dev eventually sees the bugs,
theatrically slaps his/her own forehead, repros the bug and with minimal
research, fixes it. If the bug isn’t tended to for a week, this is even more
expensive because the code associated with the bug might not be so familiar to
the dev, so it would require more research to fix the issue.
It would be MUCH better if the dev tested out the feature
first with some E2E scenarios, before the checkin, or have the tester take the
changeset (using Visual Studio’s TFS, this is a “shelf set”) and do some
testing of the changes, to find the issues before checkin. Why better? Because a)
the fix will be quicker to do b) no need to enter bugs for the record, and c)
nobody need be hindered by the broken functionality of the issues, because they’re
never checked in. Oh, and d) triage doesn’t have to look at the bugs, because
there aren’t any reported bugs.
Another useful way to address this is to check in tests for
the feature at the same time that the feature is checked in, which means that
whoever wrote the E2E tests (probably a tester) combines that changeset with
the product feature change. This can save a lot of churn, and the symmetry of
checkin in the combined feature and quality tests looks simple and robust. The
problem is if the feature automation is not ready when the feature is, and
checkin of the feature would be held back. That might slow the dev down and for
a complex product, there are likely other stakeholders (dev, test, and PMs)
waiting on the changes, so the cost of waiting must be compared to the value of
doing a unified dev + test checkin.
Therefore, the dev should be expected by team convention to
do some amount of E2E testing of a new feature. How much?
For simplicity of argument, assume for the moment that
nobody from Test is involved before checkin.
Too little testing on the dev’s part, and the dev won’t find
his/her obvious, blocking bugs. (“Blocking” means that functionality is broken
and breaks a scenario or two around the new feature, so some testing and other
use of the product is blocked.) Too much, and the feature is delayed, along
with other features and other work that depends on the feature.
I asked this question – how much testing by the devs? – of James
Whittaker, when he gave a talk last month at SASQAG in Bellevue, Washington.
James’ answer was that it depends on the dev’s reputation
for quality. Fair enough, but I’d prefer to start out with formal, uniform
expectations and relax them for individuals as they earn the team’s trust:
First, have the test team define repeatable E2E test cases
for the new feature being implemented. These test cases are to be used through
the SDLC and beyond, so might as well write them earlier in the cycle than they
normally are. Give the test cases sufficient detail that anybody who knows the
product can run them, and precise enough that distinct bugs are always
correlated with different test cases.
Then, have the devs execute the test cases when they think
the feature is ready. If the feature is non-GUI (e.g. an SDK API) then maybe
the E2E test can be implemented easily too, and the test run that way, before
checkin and then afterwards for regression. If it’s a GUI feature e.g. in a web
browser, probably the feature can’t be automated before implementation is
complete.
I recommend a minimum of two happy-path test cases, one edge
case if applicable, and one negative case. It’s expected at project outset that
a) a tester writes the test cases before the feature is implemented b) the dev
(and maybe the tester too) runs the test cases before checkin.
