Working as hard as we do, the weekend is a must-have time for relaxation. With the kids and all, I always find the time to read the weekend newspaper. 

I have a very specific order in which I read it. First, I start with the puzzle magazine. Accompanied by a large cup of coffee, I start with some number puzzles: Hidato, Kakuro, and Sudoku. Then I do some nonograms, and that concludes the part that I like in the puzzles magazine. 

Then I move to the newspaper weekend insert. It also has puzzles in it, at the back. So I start going over the insert, from the end backward. I don’t only do the puzzles there, but actually read the whole insert back to front. Why? Because it’s always structured so that the heavier stories are at the beginning – the research articles, the important topics and revelations, and as you move towards the end, the articles become lighter: interviews with celebrities, lifestyle pieces, and so on. Dealing with important stuff during the week, I like to start my weekend with the lighter reading, as superficial as it is 🙂 

There is one thing, however, that I always leave for last: the brain teaser of the weekend insert. It’s only after I finish reading the entire insert that I get to the brain teaser. And there is another important rule: I never start the current week’s teaser before I’ve finished the previous week’s one. Often, you can find a small pile of newspapers by my bedside since it takes me a while to finish everything. Of course, when I am busier, the pile gets higher, since I have less time for reading.

As I’m going through exciting changes in my business – opening CPO Bootcamps more frequently and adding other programs (details soon), it is important for me to make sure I’m maintaining a decent work-life balance. Bringing product thinking to life itself, I thought of a metric to represent that I still get it right. One option was the size of the newspaper pile by my bedside. It makes sense, doesn’t it? If I don’t finish newspapers quickly enough, the pile will grow, so if it’s too high, it means I have a problem. But that’s not always the case.

When I have loads of work, the pile will grow, no doubt. However, having a high pile doesn’t always mean that I was busier at work or at anything else. Looking at it for a while, I realized that when I have a TV series that I like, my recharging time goes to that instead of the newspaper. I also recently started doing the 16×16 sudoku which takes almost the entire weekend to solve, so I don’t get to the other puzzles and weekend reading. So if I use the pile size as a proxy to my well-being, that metric would be skewed. 

While I’m still looking for the perfect metric (and as I always say, data can’t tell you everything, so I might just rely on my gut feeling instead), here is a 4-step method to make sure that the metrics that you define for your own product are the right ones.

Step 1: Clarify the Question You Want to Answer

Since metrics are here to answer your questions, in order to define the right metrics you must understand which specific questions you want to answer with them. It sounds simple, but as you get into the details you will realize that it’s not as trivial as it may sound.

Last week, one of my customers consulted with me on what is the best metric to trace for the migration from an old system to a new one. When I asked him what is the question he is trying to answer, his first response wasn’t even a question, and surely wasn’t very specific. It was something like “I want to understand how the migration is going”. While this is a great starting point, it can be interpreted in so many different ways: it could mean, for example, “do people like the new system” or “do people have problems with it”, but it can also mean the very different angle of “how long would it take us to migrate everyone” and “is the move happening fast enough”.

The answers to these questions would be very different from each other, so you better know in advance which specific questions you want to get answered. If your immediate response is too generic as in my example above, ask yourself what about it is important for you, and what specifically would you like to know. Keep doing that until you understand exactly why this is the right question, as this process reveals your entire thinking process and helps you understand what is truly important for you. 

Step 2: Define the Ideal Metric

Once you know what you are asking, start seeking the answer. One of the most common pitfalls that I see when people come to define their metrics, is that they start by thinking about what they can measure, not what they want to measure. These are very different points of view, that would bring you to very different definitions at the end. While we always want our metrics to be practical and of course measurable, limiting yourself to what’s feasible right away won’t allow you to understand what is it that you really want and which compromises you are willing to make to be rooted in reality. Another advantage of thinking about the ideal metric first, even if it’s not a feasible one, is that it brings out the essence of what you are looking for.

In the customer example from above, this conversation happened right before the launch of the new system. The next time people would log in to the system, they were going to get a popup suggesting that they try the new system instead of the current one. If they don’t like it they can always come back to the old one using a dedicated link. When we started our conversation about the metric, the customer described to me that he would be counting clicks on the “use the new system” popup, because that was the only realistic option. But when we had the ideal metric discussion (I always ask – if you had all the information in the world and could measure anything, what would it be), we realized that it’s not actually about the clicks. The popup is just the first step of a journey to adopt the new system, and what he really wanted to know was how many people really adopted the new system, which might translate into a metric in the form of using it consistently over time. That, too, requires further clarification and refinement to fit what can be done, but as you can see it puts us at a very different starting point for this discussion, compared to the original option of measuring clicks on the popup.

Step 3: Create a Realistic Metric Formula

Metrics are eventually formulas. Someone (engineering, BI, product analysts, or product managers) would need to know how to calculate the number that tells you the current metric value. Usually, when I ask people for their metrics they give me the metric name (e.g. we are measuring retention). That’s great for a general conversation but not for the actual definition process. 

This step has actually two separate parts: first, take the ideal high-level metric that you defined, and create an explicit formula to calculate it. For example, if our ideal metric is how many people are using the new system consistently over time, the formula could be something like this: a user is counted against the metric if over the last month they used the system at least 3 times, and all of these were in the new system. They need to spend at least 5 minutes each time on the system, while actively doing work there and not just having a stale window open. 

As you can see, this level is much more detailed and well defined than anything we previously talked about. Once you translated your high-level metric into a formula, you can start making it realistic and measurable by understanding which compromises you need to make. For example, maybe you can’t really measure if they are truly active. So you will have to make do with sessions longer than 5 minutes, but perhaps add a limitation of up to 1 hour since beyond that it is highly unlikely that they are still using the system. Doing so not only helps you with the definition itself but also gives you a much deeper understanding of the metric you are eventually going to have. When you see the numbers on the dashboard, you will immediately know if they are skewed because of the compromises you had to make or if that’s just reality.

Step 4: Unit-Test Your Metric

It is usually relatively easy for us to come up with metrics that are correlated to the questions we want to answer. But correlation doesn’t necessarily mean causation, or put in simpler words – as in my newspaper pile example from above – there could be other reasons for your metric to behave the way you want, ones that would not arise from the phenomena you want to measure. So your metric might be doing well while your reality is not so much, and vice versa.

To prevent that as much as possible, ask yourself what could skew your metric that way. Some of these things you will know and notice only after the metric is already guiding you, and you see some odd phenomena. For example, at eBay, one of the categories was so huge that any phenomena happening there completely skewed any related metric, so we ended up defining two sets of metrics – with that category included and without it. But many times even just asking yourself what can make the metric behave “the right way” without a good justification would uncover pitfalls that you can avoid in advance.

As you can see, metrics, like products, require iteration and adaptation to become great at what they are meant to do. Make it a habit to always check if the metric still serves you, and improve as you go. Don’t let reality limit you, create the reality you want.