Predictive and Reflective Development Metrics

We spent some time this week working on Development Metrics. Without getting into all the details, I did want to share one illumination that came from those discussions.

Two Types of Metrics

What I realized when looking at the various metrics and methodologies out there is that they fall into two main buckets I'm currently calling "Predictive" and "Reflective." (If there are better industry-standard words for these, feel free to let me know.)

Predictive metrics are the measurements we put in place to gauge what we think is going to happen. Almost all code testing (Unit testing, interaction testing, etc.) provide us with predictive metrics.

Reflective metrics measure what actually happened.  All production monitoring (errors, performance, etc) fall into this bucket.

A Few Reasons to Care

Ok. That's great. Why does it matter?

I noticed is that this distinction is very important when communicating to people less technical, especially if those people are decision makers. I'm you've heard something to the effect of: "Why did the website break if we have all these tests?" A predictive/reflective vocab can really help this conversation stay high level instead of diving into the depths of the short falls of testing.

In addition, I think it can help decision making about where to utilize resources. When you look at resources in this light, it makes you realize how much effort you need to be spending on reflective metrics. Not that predictive metric don't mean anything, in fact I think that every reflective metric you care about needs to have a predictive process to guard it, but you may find your priorities lie in meeting your predictive metrics which may only be telling half the story. (And a wishful thinking version of the story at that.)

Even if the two above aren't a ton of help to you, this 3rd realization kind of blew my mind. When looking at "feature development and deployment" it seemed pretty clear that this is a predictive process. You think you know what a user or client is asking for. You think you know how a feature will be utilized. These are predictions.

On the other hand, user feedback and requests are reflective measurements. They let you know how things are actually being used and working and not working.

Here's the kicker. How much time do you spend in feature meetings trying to hash out the right functionality? And how much time do you spend talking to users, getting feedback, and following up on feedback? Hmmmm......

Would love to hear you metricize...