Second in a series. In part one of this article we covered how to create and manage purchase requirements. In part two, we describe a way to evaluate products against those requirements, to find the one that best matches your needs.
One way of evaluating products is to rate how well they meet the individual requirements, and express that rating as a numerical score. Tally the individual requirement scores to calculate a product score, which is a single number that expresses how close a product is to your requirements. Rank products based on these scores, and the highest scoring product is the one that best matches your specific requirements.
Finding Products
While you usually have a few potential products in mind when starting an evaluation, it is a good idea to find others that may be worth evaluating. Online competitive reviews are a very useful source of product information, and a great place to start. Try doing a Google search on the products you do know about along with words like “review” and “comparison”. This can lead to reviews that include unknown products. Also, look at any reader comments because they can often alert you to new products not mentioned in the reviews.
Asking industry peers is another excellent source of product information. Other useful sources are communities like LinkedIn groups, Spiceworks etc. Again, reader comments below articles in technical publications can provide valuable nuggets of information.
Rating Products
To rate a product you measure how well they meets each particular requirement in turn. Ratings are more than just a simple “yes or no”, and should have something like the fields listed below. Arrange these on the evaluation spreadsheet as columns under each product.
Requirement Rating: Your measure of how well a product meets a particular requirement. See Requirement Rating List below.
Rated by Optional: If there are several people rating products, you can record who rates each requirement. Typically implemented as a drop down list of names.
Rating method: A drop-down list that describes how the product was rated. For example, this could be based on the product’s webpage, you could have seen a demo, or it could be a hands on test of the product.
Requirement Score: The requirements importance has a value associated with it, as does the requirement rating. The requirement score is the product of these two numbers, and is a calculated field.
Rating Comment: This is a field where you can comment on how well the product met a particular rating. This is particularly useful when you are looking closely at the final product candidates.
The Requirement Rating is usually implemented as a drop down list with three columns:
Value: A numerical measure of how well the product meets a particular requirement.
Rating: A textual label attached to the rating value. In practice it is much easier to pick from the rating list than to pick a number.
Description: An explanation of what the rating value means to ensure everybody is on the same page.
|
Example of Requirements Rating List |
||
| Value | Rating | Rating Description |
| 0 | Does not meet | Product does not meet the requirement at all, or the feature is completely missing. |
| 1 | Slightly meets | Product has the required feature, but serious deficiencies exist in the implementation that can’t easily be worked around. |
| 2 | Partly meets | Product has significant deficiencies in the feature, but they can be worked around with some effort. |
| 3 | Mostly meets | Meets the requirement to a large extent. Deficiencies can accommodated with minimal effort. |
| 4 | Fully meets | Adequately meets the requirement. No compromises are required. |
| 5 | Exceeds | Does substantially more than is required. There is room to grow into this requirement, and there is a reasonable possibility of using the extra functionality in the future. |
| 6 | Far exceeds | Does an order of magnitude more than is required, and there is very little possibility of that functionality ever being used. Typically this indicates a mismatch between the requirement and the product being considered. Used to flag products where you would be paying for features that will never be used. |
As you work through rating different products, you always uncover new requirements. Make sure you add them to your evaluation. This is one way of ensuring you capture all your requirements.
Comparing Products
As you rate products against requirements, they generate requirement scores that are tallied up into product scores. This number is compared to a reference product that “fully meets” every requirement and the product score is expressed as a percentage of the reference product score. If you collected requirements into groups, you can weight those groups and factor this into the product score. Typically, all groups get a default weight of 1.0, and this is adjusted up or down as required.
Properly evaluating products is significant work, and it is useful to have an idea of your progress. You can see this by calculating the number of requirements rated as a percentage of the total requirements. Our experience has been that once you have evaluated over 50 percent of the requirements, the product score tends to be fairly close to the final score. However, for your short list products you should rate at least 90 percent of the requirements to ensure an accurate evaluation.
You can make product scores more useful with a “projected score” that is calculated by comparing the rated product to the reference product using only rated requirements. This estimates the product’s final score, and the accuracy improves as you rate more requirements. The projected score lets you compare products without rating them against every single requirement.
You can think of the product score as its GPA, and it gives you an excellent idea of how close the product is to your specific requirements. As a rough guide you can exclude products that score less than 50 percent as being unsuitable, while products that have an 80 percent to 90 percent score are usually very good candidates.
Occasionally you might find some products that score more than 100 percent; typically these products cost more than you want to pay.
Bear in mind that the primary purpose of product evaluation is to find the product that best matches your specific requirements as fast and easily as possible. While you are unlikely to find the perfect product at the price you want to pay, with this process you can find the best product for the price you do want to pay.
Final Thoughts on Reducing Purchase Risks
Typically, the first version of an evaluation is based on product information from the web. You can improve your evaluation quality by selecting important and higher requirements (see Requirements Importance table in first article), and testing the product against those requirements to verify your initial ratings. Sometimes there are surprising changes. You can manage this with the “Rating Method” column.
Using a structured approach to selecting products speeds up the entire process, and reduces purchasing risks. While speed is always important to your business units, watch out for impossible deadlines. These can force you to take short cuts evaluating products that can later prove disastrous.
Any significant purchase needs several reference customers. If the vendor’s reference customers will not say anything bad about the vendor, that is a warning sign! If vendor is not well known, it may well pay you to visit them in person. Are they what you expect? Even if you do visit them, make sure they are what they appear to be. For example, it is all too easy to get a tour of a data center and come away impressed, when all the vendor has is a small cage inside the data center. Ask to visit their business offices as well. If they resist the effort that is another warning sign.
Conclusion
In these two articles, we have described a structured way to create and manage a comprehensive purchase requirements list, and how to evaluate products against these requirements. The result is a list of product candidates ranked by how well they meet your requirements.
The primary purpose of product evaluations is to find the one that best matches your specific requirements. While you are unlikely to find the perfect product, you will cut through aggressive marketing and sales pitches to find the product that works best for your specific requirements. You will do this much faster than with unstructured methods, and you will be a lot more certain that your final choice was the best. By using a structured approach to evaluate products systematically, you also calibrate your own expectations. When you make the actual purchase you know exactly what you are getting, and there is no buyer’s remorse.
Photo Credit: Dusit/Shutterstock
Chris Doig has personally seen the problems caused by poor technology purchasing in multiple companies. He co-founded Wayferry, a startup that created a free decision support tool for technology purchasing. Wayferry’s mission is to help IT people everywhere make better technology purchasing decisions.