How to Get the Most Out of an Evaluation Matrix
“In my view, evaluation matrices are one of the most misused, misunderstood, tools of product development. However, a simple shift in the way we think about them can help us all get the most out of evaluation matrices. ”
After generating a number of concepts to satisfy a particular design problem, it’s worth pausing to identify the strengths and weaknesses of each concept – or parts of concepts. This is a powerful step leading to the creation of new concepts that combine the strengths of concepts already generated. Although evaluation matrices are primarily for this purpose, few people use them to help create these new and improved concepts. Instead, evaluation matrices are often used as decision matrices aimed at helping teams decide which one of the generated concepts to choose. When used in this way, evaluation matrices lose most of their value as a product development tool.
Fig. 1 - Basic structure of an evaluation matrix.
What is an evaluation matrix?
An evaluation matrix [1] is a simple table with evaluation criteria listed in rows, and design concepts listed in columns. In the body of the matrix, each concept is evaluated against each criterion. Through a side-by-side evaluation, strengths and weaknesses of the different concepts emerge and room for improvement becomes more obvious.
The evaluation matrix is a fundamental part of what its greatest proponent Stuart Pugh (1929-1993) called controlled convergence, and which he described as “Concept Selection: A Method that Works.” The beauty of the controlled convergence method is that it allows concept generation (divergence) and concept selection (convergence) to be meaningfully carried out iteratively and synergistically.
To make this important point more clear, first imagine the situation where controlled convergence is not used. Here, concept generation and concept selection are carried out back to back, in one go. Under this approach, 10 ideas (for example) are generated, and then an evaluation matrix is used to score each idea in order to pick one of the strongest ideas. Notice that this underwhelming approach is decidedly serial – generate concepts, then select one using a matrix. When used in this way, matrix methods fail to live up to their potential. The truth is that this is how most people use evaluation matrices.
Fig 2. - Diverge then converge approach.
Now imagine the situation where controlled convergence is used. Under this more effective approach, concept generation and concept selection are more intertwined. Here, 10 ideas are generated and an evaluation matrix is used to identify the strengths and weaknesses of each design. Then the strong parts of one concept are combine with the strong parts of another concept to create an entirely new concept, stronger than the two it originated from. Now imagine doing this three of four times in a row. The concepts generated in the final round will be noticeably stronger than those generated in the first round. Recognize that evaluation matrices are a fundamental part of this process. Unfortunately, almost no one uses evaluation matrices in multiple rounds like this, which of course limits how useful they are in practice.
Fig 3. - Diverge-converge cycle used in multiple rounds.
How did such a great tool fail to live up to its potential?
Most of us were first introduced to evaluation matrices in a university setting. They were taught as selection matrices [2], screening and scoring matrices [3], weighted matrices [4], or decision matrices [5]. Introducing evaluation matrices in this way has at least two significant problems.
The first problem is that the name of each of these methods suggests its purpose is to help converge upon, select, pick, or decide on a concept to “go with”. This is almost in direct opposition to the greatest strength of the evaluation matrix, which is to facilitate subsequent rounds of concept generation by helping designers focused on creating new ideas that build on the strengths of previously generated ideas. Frankly, it is limiting to think of an evaluation matrix as a “decision matrix” or a “selection matrix”. Matrices don’t make decisions or selections.
The second problem is the fact that selection matrices are often taught in passing, in a university setting. Both professors and students have contributed to the problem (I have been both, so I can say this with some degree of realism). To understand this, first recognize that evaluation matrices are one of the few conceptual design tools that could have some math associated with it. To be blunt, anything with math is easier for the professor to grade than things without math. This has caused some professors to overemphasize the calculations associated with evaluation matrices, which have indeed turned evaluation matrices into scoring matrices, weighted matrices, and decision matrices. Additionally, we know that students are excellent at optimizing their time. If they’re taught a tool that benefits from multiple rounds of use, students will fail to see the benefit since they will generally carry out as few rounds as possible to complete an assignment. For evaluation matrices, that would be one uninspiring, ineffective, round.
Tips for getting the most out of evaluation matrices
If you want to get the most out of an evaluation matrix do these things:
Recognize that its greatest value is to identify the strengths and weaknesses of concepts. Not to calculate concept scores.
Use the evaluation matrix in an iterative cycle of multiple rounds of concept generation/concept evaluation.
When filling in the body of an evaluation matrix, make simple evaluations of each concept relative to benchmark concept (better than, same as, worse than). This benchmark could be the market best or any good concept in the set of generated ideas. When it makes sense, move to more complex evaluations (much better than, better than, same as, worse than, much worse than).
To prevent gamesmanship during the evaluation process, it is helpful to evaluate all concepts against a particular criterion before moving to the next criterion.
Appreciate that there is significant value in filling out the body of an evaluation matrix in a team setting. The discussion that accompanies the work will help clarify what the concept under evaluation actually is, and allows for varying points of view to be expressed regarding a concept’s strengths and weaknesses.
[1] Pugh, S. (1981, March). Concept selection: a method that works. In Proceedings of the International Conference on Engineering Design (pp. 497-506).
[2] Eppinger, S., & Ulrich, K. (2015). Product design and development. McGraw-Hill Higher Education.
[3] Mattson, C. A., & Sorensen, C. D. (2020). Product Development: Principles and Tools For Creating Desirable and Transferable Designs. Springer, Cham.
[4] Cagan, J. M., & Vogel, C. M. (2002). Creating breakthrough products: Innovation from product planning to program approval. Ft Press.
[5] Mullur, A., Mattson, C., & Messac, A. (2003, January). Pitfalls of the typical construction of decision matrices for concept selection. In 41st Aerospace Sciences Meeting and Exhibit (p. 466).
