Verification Versus Validation
The product development process transforms an idea from something vague and ill-defined, to something finished and desirable. That transformation occurs as we do design activities that cause the design to become better [1].
An often overlooked truth of product development is that the first version of the design will not be the best design [2]. It is that first version, however, that helps us see the problems that need to be taken care of, and the avenues for improving the design. That first version also helps us see what’s working well with the design. The approaches we take to discover what’s good and what’s bad about our design has a significant impact on how slow or fast the design evolves.
There are two general types of tests that can be done to observe the design’s strengths and weaknesses. They are called verification tests and validation tests.
The differences between these tests can be confusing because the words seem and sound similar, and because some people use them interchangeably without thinking deeply about what they mean. This article describes the difference between them. To be clear, the essence behind the following definitions is not mine. It is from the Project Management Institute, which is a well-known entity that drives project management in many fields and has for more than five decades [3].
Verification Tests and Validation Tests
Verification tests are those done internally by the product development team to see if the product functions the way it should.
Validation tests are those run externally to the team, to see if the product is desirable to the market/stakeholders, and to see if the product meets their expectations.
To further illustrate, imagine that I am designing a backpacking stove, and it needs to weigh less than one pound. Throughout the development process, I will use verification testing to see if I am meeting the one-pound requirement. Early in the process, before I have built anything, a CAD model with assigned materials will allow me to carry out a mass analysis to check to see if the product weighs less than one pound or not. If the mass properties analysis indicates that the product is too heavy, I’ll change the design, re-run the mass analysis and check to see if I’ve made an improvement. Every time I check, I’m doing a verification test. I’m verifying if the design functions or does not function the way it is supposed to.
If one of the stove requirements is that its set-up be intuitive and easy for a backpacker to use in the snow, then validation testing is what we turn to. Importantly, with validation testing, the one making the measurement, or doing the evaluation, or making the assessment is NOT someone on the team. It is someone from the market, in this case an actual backpacker or set of backpackers in the snow. Why is the team or a team member not a legitimate evaluator in this case? They are simply too close to the product to judge it objectively. They know too much about the product. How it should work, how it should be opened, levelled, started, used, collapsed, and stowed. Because they have created the product being evaluated, they have too much love for the product to see its flaws. As soon as the team puts the product in the hands of someone in the market, and feedback is sought, insight that could never be gained from the team alone will begin to emerge. That kind of test is validation testing.
The primary difference between verification testing and validation testing is the assessment instrument. For verification tests, the instrument is usually some sort of measurement device, such as a tape measure, a scale, a gauge, or a computer simulation, while the instruments used for validation testing are the people themselves from the market/stakeholders. Did they like it, how did they react to it, what was their body language, what did they say about it, did they buy it, would they buy it, how much would they pay for it? These are the kinds of findings that result from validation testing.
How to get test results that can be trusted
For both types of tests it is important that we use a variety of test samples. For verification testing, the use of multiple test samples allows us to understand how variation in material properties, manufacturing process, and assembly affect the performance of the product. For validation testing, it is also important to test the product under a range of conditions and not just under ideal operating conditions [1]. This could be achieved, for example, testing backpacking stove set up at a picnic table in the calm environment of early summer, and at a high-altitude, snow-filled, windy environment.
For validation testing when we’re seeking feedback from the market, we want to survey a representative set of people from the market, that span the potential users. For the stove, this may mean testing the stove with expert versus novice backpackers, with male and female users, older and younger users, and potentially with English-speaking users versus non-English speakers.
One of the main things to consider when doing validation testing is that we want to elicit an honest response [4]. There are a few ways to do that.
Do the validation testing with people who don’t have an interest in you or the product. So, not with your friends, not with your family, not with your neighbors. These kinds of tests are likely to involve people who are less willing to tell you what they actually think.
Encourage an honest response. Let them know that their honest response is important for your research, and that you’re in the middle of the product development process and expect to find things that are weak and need improvement.
Be sure to not lead the person to say what you want them to say. One of the best ways to do this is to hand them a product (prototype) and simply observe what they do with it. Do they know how to hold it, to open it, turn it on, to use it? Do they use it as you expected? What does their body language tell you? Are they excited or confused? Are they not sure what the product is all about, or do they get it? Letting a person’s first interaction with the product play out in this way is much more meaningful than first giving them a tutorial, and explanation of the product’s features. If the product gets mass produced, you won’t be with every new customer when they first see and interact with the product, and if you’re expecting someone to read a user’s manual or quick start guide before messing around with the product, you’re fooling yourself.
Why Verification and Validation Tests Matter
Simply stated, verification and validation tests are essential to successful product development.
For verification testing, it is useful for the team to recognize that no one else besides them is going to do verification tests until the product gets on the market and people find out for themselves if the product works the way it should. Therefore it is the responsibility of the team to make sure that verification testing is done and the product is certified (so-to-speak) before the market/stakeholder is expected to trust it.
For validation testing, it is critical for the team to recognize that because of their understanding of and love for the product, they cannot see its flaws the way the market/stakeholders will. Without engaging with the market/stakeholders through validation testing, the team is designing in a vacuum [5], and less likely to produce a product that meets market/stakeholder needs.
Closing Thought
I know of no good product that ultimately ends up in the hands of people that did not involve both verification testing and validation testing. It was essential for those products to be verified by the product development team as functional. It was also essential for the product to be validated by the market/stakeholders as meeting their subjective expectations and needs, and validated as being desirable to them.
References
[1] Mattson, C. A., and Sorensen, C. D., Product Development: Principles and Tools For Creating Desirable and Transferable Designs, 2020, Springer, Cham.
[2] Mattson, C., “Iteration: The most important concept in design,” BYU Design Review, 2020, https://www.designreview.byu.edu/collections/iteration-the-most-important-concept-in-design, accessed 10 Nov 2020.
[3] Wikipedia, “Project Management Institute,” https://en.wikipedia.org/wiki/Project_Management_Institute , accessed 10 Nov 2020.
[4] Ulrich, K. and Eppinger, S., Product design and development, 2015, McGraw-Hill Higher Education.
[5] Kelley, T., The Art of Innovation: Lessons in Creativity from IDEO, America’s Leading Design Firm, 2001, Doubleday, New York, NY.
