January 6th, 2010GQM: Metrics come last
Inspired by a post on the Lean Blog, Pat reminds us that you can’t measure everything effectively. Having written my Master’s thesis on metrics for Agile projects, I’ve learned and read about this in a lot of different places. One approach that is very known to Empirical Software Engineering researchers is the Goal-Question-Metric approach, first published by Vitor Basili et al. in the 90’s.
The GQM model suggests a hierarchical view of three levels to define which metrics to use:
- Conecptual level (goal): the motivation for measurement. Measuring things without a purpose and a thorough understanding of the problem will lead to meaningless metrics. This level imposes the hardest questions: what’s the purpose? what’s the object of measurement (your product, process, people)? what’s the motivation? who is interested in this goal? what are the quality attributes?
- Operational level (question): at this level, a set of questions are defined to try and correlate the object to the quality attributes we are interested in. These questions should help in understanding and assessing the current situation, but also in identifying ways to determine whether the goal is achieved.
- Quantitative level (metric): only then a set of metrics is associated to the questions, to try and find a quantitative way to measure and answer it. These can be objective (like code coverage), or subjetive (individual’s ranking of current code quality). Finding these metrics is not easy either.
It’s easy to try to cut corners and get into the things that are easy to measure first, specially when you can collect lots of quantitative data to work with these days. However, if you you don’t stop to think about the goals and motivations for measurement, it’s easy to forget the systemic complexity that surrounds us and look only for the easy-to-track numbers.
Lean management and problem solving is known for taking a very thorough and detailed approach in the understanding phase. To many people this is a paradigm-shift approach to management. Don’t let the numbers fool you, use them to your advantage.
Twitter
LinkedIn
Facebook
Flickr
Github
January 6th, 2010 at 1:53 pm
Thanks for the link. As someone who is not a developer, I have a question about how you measure code “quality.” Number of bugs and number of defects would seem to not go far enough. How do you measure fitness for use? Seems like a lot of that would be subjective, but it has to be managed anyway, correct?
Can you share an example of something you manage, but cannot be measured?
You’re right that the things that are easy to measure often aren’t the important things.
mark
January 7th, 2010 at 6:25 pm
Mark,
There’s usually two ways to look at software quality: intrinsic quality and extrinsic quality. Intrinsic is what makes sure the software runs as expected (performance metrics, test coverage, how long it takes to deploy, code complexity, size, etc.). Extrinsic is what makes sure it’s fit for purpose, which is a lot harder to measure. It’s usually from the point of view of the customer (are people using it? how many complaints? are people paying for it?)
But, as you said, this leaves out a lot of quality attributes that should be managed but can’t be measured, like: usability, gameplay (if you’re building games), customer satisfaction, team/employee satisfaction, …
Regards,
Danilo