Project Metrics: The Double-Edged Sword

April 14, 2012 | Author: PM Hut | Filed under: Procurement Management

Project Metrics: The Double-Edged Sword
By Pam Stanton

We all know that metrics are critical to running a business. We need those leading and lagging indicators of performance, so we can change course where needed and keep doing what works.

Metrics, however, are a slippery beast. In addition to measuring the results of actions, they also motivate action. And this is where things can go either fabulously right, or terribly wrong. What’s worse is that we may not realize what outcome we are promoting until it is too late.

Here’s an example from early in my career. This experience seared onto my brain the awesome power of metrics to influence behavior. And with that power comes great responsibility to ensure that the outcome we motivate is the one that we want.

I was working at a large financial services firm as the relationship manager to the outsourced IT vendor. In this arrangement, the vendor would prepare financial estimates for each system change request from us, the customer. These financial estimates were then used to secure a budget to complete the work.

We had a huge problem. The financial estimates varied wildly from the actual project costs, with swings of plus-or-minus 75%. This caused heartburn for the business owners, because they couldn’t predict whether their funding requests would be wildly overstated or understated. I could understand the problem of being over budget, but I didn’t get why they cared so much if they came in under budget. That’s saving money right? Wrong! It was then I learned that coming in significantly under budget is just as bad, if not worse, than coming in over budget—because you have tied up company dollars that could have been allocated elsewhere.

My division’s Vice President had a solution—or rather, a demand. We would impose a new metric immediately that required the vendor to deliver each project at no more than plus-or-minus 10% of the original estimate. Failure to meet this metric would result in a financial penalty to the vendor.

Within 3 months, we had a perfectly green scorecard. It was amazing! All we’d needed all along to keep that vendor sharp was a metric! Eureka!

Or not? Even though I was still early in my career, I smelled something funny. I wanted to understand this power of the metric, and how it had changed everything about the vendor’s performance. Were they just now being more careful? Did the old “What gets measured gets done” adage apply? I had to know.

I kept poking around about this measurement. Finally, I got the answer I sought from one of my vendor counterparts. His terse revelation went something like this.

“Well, we’ve always gotten really vague requirements from the business side, and we still do. We always tried to do our best on the estimates, given limited information and broad assumptions. Now that we have to meet this metric, here’s what we do. We pad all the estimates to make sure we won’t be over budget. And then if it turns out that we’re running more than 10% under budget, we just pad more hours onto the project until we’re within the 10% tolerance.”

Ouch! We had gotten what we’d asked for. All projects were delivered within a 10% budget tolerance. Turns out that all projects had also become more expensive. Oops. We weren’t measuring that…

This lesson has stuck with me throughout my career, and I call on it whenever creating performance measures. It’s critical to analyze what behaviors will be driven by the measure, and how they may actually be counterproductive. It’s still true that “What gets measured, gets done.” The trick is to make sure that what gets done is what we really want to get done.

Pam Stanton is the owner of Heart, Brains, & Courage, LLC, the parent entity for “Pam Stanton, The Project Whisperer” and “Perspectives Gallery.” She has 25 years experience leading business initiatives of all types, focusing in corporate I/T.

3 people have left comments

Pam, I love this story. True is your point about metrics… you get the results of what you measure. So you must be careful about what you measure!

I get frustrated by the notion that we should measure projects on schedule, budget, and scope (the traditional triple constraints). A project can meet all three and still be a massive failure. I’m not saying you should throw out those measures… but they should be balanced with measures from the business… revenue generation targets, operational efficiency targets, customer satisfaction, etc.

Thanks for a great post!

Thomas wrote on April 15, 2012 - 8:32 am | Visit Link

I agree, and this is a huge problem in the State budget. The projects must have a lower cost but if the budget are not spend next year their funds will be cut and people will think that they did not work.

Guilllermo Viale wrote on April 15, 2012 - 9:20 am | Visit Link

Funny is that why we have to make a fixed budget and set a stupid tolerance ratio on it to make projects equally costy.

Anson wrote on April 18, 2012 - 2:20 am | Visit Link

feel free to leave a comment

Comment Guidelines: Basic XHTML is allowed (a href, strong, em, code). All line breaks and paragraphs are automatically generated. Off-topic or inappropriate comments will be edited or deleted. Email addresses will never be published. Keep it PG-13 people!

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

All fields marked with " * " are required.

Project Management Categories