Agile testing is not about automation

I’ve touched on this subject in a previous post, but this time I want to dive in head-first in response to the following quote from a recently released study on Agile Testing:

“Because the ultimate goal of Agile testing is to automate more effectively…”

Although taken slightly out of context, the message conveyed by both this statement and the study as a whole is clear: the holy grail of Agile Testing is test automation. As much respect as I have for the authors, I have to disagree. Test automation can surely play an important part in an Agile Testing strategy, but the ultimate goal has nothing to do with automation, exploratory testing, or any other QA practice. The ultimate goal of Agile Testing (for me at least) is to make testing (and a QA mentality) an integrated natural part of the agile development process used at hand, be it Scrum, Kan-Ban, Lean or any other.

I guess this is still somewhat high-level, so I’ll try to be more concrete.

To me, the most important aspects of Agile are those of collaboration and communication; cross-functional teams working closely together, preferring interaction to documentation. To succeed, quality has to be an integrated part of this process. QA engineers are essential participants in the very first discussions, at sprint planning meetings and at the daily standups. They continuously scrutinize the product to assess quality issues and plan testing activities proactively, be it with UX designers (usability testing), developers (code reviews, TDD/BDD, unit-testing) and operations (load and security testing).

The agile testing quadrant created by several prominent persons in the agile testing community (see for example Lisa Crispin’s blog) gives a good overview of common testing practices and how they come together in an Agile project:

(my own version based on Lisa Crispin and Michael Hütterman’s version)

Moving from left to right, testing activities go from supporting development (i.e., checking that stuff works) to critiquing the product (i.e., trying to break stuff). From bottom to top, tests are technology-facing (i.e. testing at the technology level) to business-facing (i.e. testing from the user perspective). Please note that none of the testing practices named in these quadrants are specifically “Agile.” It is when you apply them in an Agile project that your testing becomes Agile.

So what about automation? First, a quick definition (from Wikipedia): “Test Automation is the use of special software (separate from the software being tested) to control the execution of tests and the comparison of actual outcomes to predicted outcomes.” The main goal of test automation is usually to run a (possibly very large) set of tests repeatedly to make sure your software doesn’t break because of introduced changes. Since most agile development teams adopt the practice of Continuous Integration, automated tests are usually executed as part of the continuous integration build process.

As you can see in the above quadrant, automation could definitely cover those QA/testing practices that are technology-facing and that support the development team (i.e. make sure things work as expected), including unit and component tests, TDD and BDD. Automation then extends to some degree upwards (to business-facing tests, for example automated UI Tests) and rightwards – to tests that are targeted at finding “holes” in the product – such as automated load tests and production monitors.

Automation definitely does not make sense where human expertise, intelligence, and assessments are key. Exploratory testing, usability testing and overall acceptance testing are primary candidates for manual testing, as are some aspects of (exploratory) load and security testing, as these tests all require “out-of-the-box” thinking by the tester to provoke the product and identify inconsistencies and irregularities that are impossible to code for in an automated test.

If you buy into the premise of the Agile testing quadrant, only roughly half of the mentioned testing practices are applicable for automation – and these are primarily technology-facing tests. Aren’t the business-facing tests just as important? I would definitely argue that they are, especially for assessing if you are actually building the right thing for your users. It doesn’t matter if the smallest pieces of your system work well if they haven’t been assembled into something your users want or need.

Another challenge imposed today on many Agile development teams is the desire to shorten release cycles from months to weeks to even days or hours. How can you maintain reasonable testing when this happens? Is it more important to do usability testing or invest in more automation? The answer is difficult and product-specific, and this is where Agile shines. By making these decisions collaboratively within a cross-functional team, you have at least maximized the probability of coming to a sound conclusion.

So the moral of the story: automation definitely has its place in Agile Testing, but it’s far from the silver bullet some would like you to believe it is. Use it wisely and it will be a great foundation for your testing strategy, and, more importantly, be sure to implement Agile Testing the way it should be done – by making testers and testing a vital ingredient in your agile process.