In the agile space, we talk a lot about testing. How will we test things? What should we test? How can we automate our testing? Yet, testing isn’t actually the point. What’s important instead, is quality. If we had amazing quality then it wouldn’t matter if we’d tested or not.
The trick, of course, is that testing is one of the ways that we determine if we have problems with quality. If we have no tests then we’re often unaware of what kind of quality we have and yet the mere presence of those tests proves nothing about whether or not we have quality.
So if quality is the real concern, why do we focus so much on the tests instead? This is because of a psychological phenomenon called attribute substitution1, where in an attempt to reduce the mental effort required, we replace a complex decision with a simpler decision and aren’t even aware that we’ve done so.
If the problem we were trying to solve was actually testing then it might make sense to have dedicated people who specialized in that. This is the kind of thinking that has led to the creation of dedicated “quality assurance” (QA) or “quality engineering” (QE) teams, and it’s actively detrimental to the improvement of quality, as we’ll see.
At this point, you may be thinking that I’m about to propose not testing at all and that’s not the case. There is good value in testing, it’s just not the value you may be assuming is there. Specifically while testing can be good at identifying when we made a mistake and quality is starting to slip, that testing did nothing to actually give us good quality in the first place. Quality cannot be tested into a product. By the time we start to test, the quality is either already there or it isn’t.
“We cannot rely on mass inspection to improve quality, though there are times when 100 percent inspection is necessary. As Harold S. Dodge said many years ago, ‘You cannot inspect quality into a product.’ The quality is there or it isn’t by the time it’s inspected.”
– W. Edwards Deming, “Out of the Crisis”
You may be thinking now, “What about test-driven development (TDD)? That’s happening before the code is written”. While that is true, TDD is only indirectly a testing activity. It’s more precisely a design activity that happens to leave automated tests scattered in its wake. Done properly, TDD drives the design of the code and does have a direct impact on some aspects of quality.
So what really is quality then?
Jerry Weinberg defines quality as “value to some person”2 and while that is fairly vague, it’s also a good description. Quality is different for each person and so to understand whether we have quality, we have to be very clear who we are creating the product for and how they intend to use it.
Quality is value to some person.2
Imagine that I handed you a coffee cup and asked if it was good quality. How would you verify that? Given that the coffee cup already exists, you can no longer build quality in to it so now you have to resort to testing. Does it already have quality or not?
You might put a liquid inside to see if it leaked. You might try a hot liquid, anticipating that I’d be putting a hot beverage into it. Would that tell you if you have good quality? Well it might if the only thing our user cares about is that it holds a liquid.
What if our user wanted it to have a personal photo on the front? Would testing only hot liquids be enough? No, we’d have to look at how clear the picture was and perhaps whether the picture would wear off during repeated washings.
We need to understand our user and the intended use of the cup before we can determine if the product already has good quality. This is the essence of Weinberg’s “value to some person”.
What if we wanted to improve the quality of the coffee cup? We might be able to add something extra at this point like extra glaze or reinforcing the strength of the cup but overall, it’s too late; the time to improve quality would have been while it was being created.
So let’s look at software teams. All too often, we leave testing to the end. The developers write some code and throw it over the wall to the testers. By the time they get it, the code either has good quality or not — there is nothing the testers can do to improve the quality except throw it back over the wall so the developers can work on it some more. This is very expensive and forces a bunch of rework when we discover that quality wasn’t good enough.
Today, at least the developers and testers tend to be on the same team, even if they aren’t actively working together. In the no so distant past, testing would be a completely different team and so the hand-offs between developers and testers would be very expensive.
If we want to build better quality then we have to eliminate that hand-off. Developers and testers need to work together while the product is still being built.
We’ll still want automated verification of the system to ensure we haven’t made mistakes, but these checks should be created while we are still actively building the system, not after we’ve finished. This is the essence of Test Driven Development (TDD), Acceptance Test Driven Development (ATDD), Behaviour Driven Development (BDD) and other techniques.
The people on our teams who have been doing the testing up to now, tend to have excellent skills that help with quality. We’re just engaging them at the wrong point in the process. They should be involved earlier, when we’re actually building the system, not afterwards when it’s all done.
Writing tests after the fact, is falling into the trap of attribute substitution. We’re not longer solving the right problem. Now we’re building tests, rather than quality.
-
Attribute substitution is a situation where, when we are faced with a computationally difficult decision, we will often substitute a more easily calculated decision in it’s place and answer that instead, without even realizing that we did this. More details, with examples here ↩
-
Jerry Weinberg describes quality as “value to some person” and gives a large number of examples of this in his blog post on the topic ↩ ↩2