Apr
02

How Should You Measure a Product Manager?

By on April 2, 2012

Rich Mironov at Product Bytes weighs in with a thoughtful post about measuring product managers.  He had a recent experience in Sweden where the product executives he worked with shared the same concerns as product executives in the U.S., Europe and the rest of the World, namely: what is the right way to measure the performance of a product manager?

This is a really interesting and difficult question to answer.  Rich breaks it down into three areas: product-level metrics (product revenue, profit, customer sat), team-level metrics (are the other teams getting what they need from product management), and individual-level metrics.  The first two metrics are fairly well defined – there is a strong history of measuring products and teams with these criteria and there are MBA programs with entire curriculum defined about measuring product performance.  Unfortunately measuring individual product manager performance is more tricky as these metrics are not as clear.

Lacking traditional or clear metrics, most companies rely on product-level metrics such as product revenue, margin, or customer satisfaction to measure the performance of an individual product manager.  The thought process is defensible – you want the product manager to be motivated by metrics that lead the product toward success.  Unfortunately, product managers typically have very little control over the outcome of these metrics, or at best, indirect control.  For a product manager measured on product revenue, the product manager could spend months designing the perfect product, training the sales channel, and working with marketing to design a lead generation campaign – only to wake up one day and find that the executive team had made a strategic decision to make a large acquisition and change Sales resourcing.  That change would have a huge impact on the product manager’s metrics, at no fault of the product manager; effectively they would be penalized for doing the right things.  For a product manager measured on product margin, I always ask “Do you have the ability to outsource the development of your product to a 3rd party if your internal engineering team delivered an estimate that was too expensive?”  For most product managers, the answer is a resounding “no.”  Customer satisfaction suffers from a similar control issue – there are lots of reasons beyond the product that customer sat may suffer, and most of those reasons are not something a product manager can impact.

The other issue with measuring product managers on product-level metrics is an issue of timeliness.  If you are measuring your team on product revenue, you can make an argument that it may take up  to 18-24 months to know if the product manager is doing a good job.  Consider this – if you hire a new product manager, before he or she can impact product revenue, the following will need to occur:

  • The new product manager will need to go into the market and research the market’s problems (up to 3 months)
  • The product manager will then write what they have learned into a business plan, and get it approved (1-2 months)
  • Then the product manager will write requirements for engineering (up to 3 months)
  • Then engineering will build something (6-9 months)
  • Then in most products there will be a Sales cycle (3-6 months)

Only then will we have the data to understand if the decisions made on the front-end of the process are “good.”  Hopefully we will be more agile and most faster, but the point remains – using product-level metrics to measure individual performance is insufficient.  We need a better way, and there is a better way.

What I have settled on for measuring individual performance is to measure the activities required for a product manager to be market-driven:

  1. Setting a quota for market visits (10/quarter is a good start IMO),
  2. Being able to defend an updated business plan in front of me and their peers every quarter, without using the phrases “I think,” or “In my opinion” (which requires higher order thought and research).
  3. Keeping their finger on the pulse of the business by defining and measuring the right product level metrics and communicating them in the form of a dashboard report monthly.

I like these metrics because they require a product manager to get out from behind their desk and be outside-in focused, and the product manager can control them, as opposed to revenue, margin or CSAT which the product manager cannot control.

All things being equal, it would be preferable to measure outcomes as opposed to activities, but it is wicked hard to separate and quantify the inputs of a product manager vis-a-vi all the other variables that go into making a product fly.  Activities are a sufficient proxy to measure a market-driven product manager to supplement product-level metrics and understand if a product manager is doing their job – without having to wait two years to find out.

Comments

  1. Evan Moore says:

    This post is super-relevant to me right now considering I just wrote my yearly self-evaluation earlier today. While I agree with your suggestions for assessing a Product Manager’s performance on an individual level, in agile organizations with tighter development cycles the time needed to asses whether a PM has been effective by looking at the performance of the product becomes much less that 18-24 months.

    For example I’ve been on a new product within my company for just about 4 months now, and while I am only now working on features that I personally researched and advocated for, the result of my work can be seen in features that I defined and managed the implementation of in just the 4 months I’ve been on the team. I would go so far to say that in an agile organization if a PM’s efforts haven’t made a measurable impact on a product within 6 months, then something is wrong.

    I’m not necessarily saying the PM has done something wrong, but either they aren’t doing enough to affect change or they’re dealing with very stubborn business owners/stakeholders.

    Great post!

    Evan Moore
    http://www.linkedin.com/in/evomoore

  2. Paul says:

    Thanks Evan, I agree with you. Ideally in an Agile shop a PM should be measurable using product-level metrics faster, although I’d contend that waiting 6 months is still too long (at least initially). Measurement is both art and science so a good manager of PM should look at both activity based metrics and product-level outcomes.

  3. lmckeogh says:

    Paul, I’ve been thinking about this on/off over the last 2 days since Rich’s original post. I came across a question that caused me to pause. The question was what do you do that if you didn’t it wouldn’t get done? (or something to that effect) Basically, what makes you irreplaceable. I think you may have said it in the past that the product manager’s job is one that wouldn’t be missed for long if s/he wasn’t there.

    With that in mind and trying to figure out how to evaluate oneself or others, I’ve been starting with this question. If you don’t have an answer for it then that should be a pretty good clue about how to measure yourself.

    The product metrics identified are fine, but all the reasons given are true in terms of how the product manager may or may not be responsible for them. Would others on the team point to the product manager as that linch pin for success? Failure, maybe too quickly to save themselves.

    Anyway, to get a better sense I’ve taken another post to heart and sought some of the answers. Fred Wilson blogged some time back about Continuous Feedback So once a quarter I get the feedback for my stakeholders and track to continuously improving. In the Agile house this ensures I never get too far from what my constituents need and I can be sure to always have an answer for what wouldn’t get done if I wasn’t there.

  4. Yuval Yeret says:

    Very thoughtful post Paul. I agree with your view here.
    I like your suggestions for what it DOES make sense to look at.

    I would add that the Lean Startup concept of validated learning / innovation accounting take your ideas further.
    They provide more rigor around measuring learning about the market and customer needs and the search for product market fit

    I believe adding questions around how tight are the feedback loops the PM runs? Is he using minimal viable products/features?

    A PM effectively using a custdev + lean startup
    Approach maximizes the chances for product success in an uncertain shifting context. So as a leading indicator it is a good complement to lagging indicators such as product success and bottom line results.

    For PMs working on next horizon products it is even more important
    ( using Moore’s Escape Velocity language )

  5. Paul says:

    Thanks Yuval. I also admire Ries’ and Blank’s Lean Startup/Customer Development thinking, and they’ve colored my thoughts significantly. I’d agree that the long term evolution of both Product-level metrics and Product Manager metrics are going to go in that direction: e.g. how fast can you iterate through a Build-Measure-Learn cycle, can you assemble and defend a business canvas, etc.

    For now, based on what I’ve seen as I’m out there teaching Pragmatic techniques – this is a bridge too far for most. There are still a lot of lightbulbs that get turned on at the realization that product-level metrics might just not be the right metrics for measuring an individual Product Manager. It’s going to be an evolution more than a revolution in measuring product people.

  6. Derek says:

    Really late on this, but don’t you find the # of appropriate market visits is really circumstantial based on the lifecycle of the product and the existing backlog? Sure, I can understand that requirement to a degree because you never know when the next major idea will strike — but let’s say you’re a SaaS enterprise product that has been shipped for several years and is in the continuous improvement mode (so it already has a fit)?

    You’ve got a backlog already of hundreds of features, be they from competitive analysis, user feedback, internal stakeholders, or support requests and you can get through maybe a dozen of them in a given quarter. Is it worthwhile to keep going to market and soliciting feedback (even if the feedback is just to inform your backlog grooming activities) and ballooning that backlog even more? Can’t that hurt, since now your customers have a degree of expectation?

  7. Paul says:

    @Derek – I would agree that the amount of visits would change as the product travels through its lifecycle. But even a mature product that has achieved product/market fit and has a big backlog of features will eventually grow stale if we just work through the backlog. It becomes even more important with a big backlog and limited resources to have fresh market data to do prioritization – what was important 3 quarters ago when the feature was put into backlog may not be important today.

    Expectation management is a different problem. Always difficult, but manageable if the Product Manager sets clear expectations up front that he/she is there for research, not promises.

  8. [...] 2: chart a plan to achieve the 1/5/10 year plan, break 1-year plan into measurable goals for the year, design a plan to “sell” the goals to protégé’s management as [...]

  9. [...] become less important. For more ways to measure your product managers’ performance right now, read the full article by [...]

  10. Narasimha says:

    I agree with most of the written context. However, there are some considerations to be taken into effect. While performance is critical, it also depends on the proper definition and Company Operational process. Every company is unique in terms of the way it commits to the businesses.

    Hence, as a PM, it is vital to apply trade-off mechanisms keeping cross-functional teams in the loop. He/She is answerable to the stakeholders and need to click at the right point of time.

    Therefore, PM alone cannot be deemed as misfit/non-performing

  11. Santosh says:

    Nice post Paul, Just one clarification; product managers are not only accountable for success of new products but also for the existing range already available for sales team and customers. How would you recommend performance metrics for product managers in such case?

  12. [...] Paul Young  and Rich Mironov shared illuminating posts last week  on how to measure a product manager. That gave me a fillip to write on the same topic borrowing and adding on to his thoughts. [...]

  13. […] is measured differently than Product Management and Development.  We teach that Product Management should be measured on its ability to be […]

Leave a Reply