Friday, May 30, 2014

Beware of scales

Linear scales and others
And so we continue exploring the problems that can arise when selecting and evaluating projects, in particular when we're real trying hard to be objective and transparent about the evaluation process. This time we'll be covering scales, that is, what is the impact that measurements (and their respective scales) have on the end result. Can it be that just changing scales can change the way we appreciate and perceive things?

Contrast effect

This article is not about the contrast effect, but it relates to it. The contrast effect is what makes you feel one hand warm and the other really hot when they are both inserted in the same bucket filled with tepid water - just because one hand (the one that feels hot) was before inserted in ice cold water.
We naturally distort our perception so things fit better in our minds: the tepid waters feels hot when comparing to the ice cold water where our hand was before. However, in this article we're going to be (again) very objective and reasonable about the scales only to find them incoherent!
So let the fun begin!

Linear scales

First off, lets check our most common types of scales - linear scales. Almost every single thing you can think of is linear when you measure it, right? Weight, temperature, height, memory: twice the memory is worth twice as much, right? If an external disk with 1TB is worth, lets say, 100€, 2 of these disks would be worth exactly 200€ just the same as 2 Kg of sugar cost twice as much as 1 Kg of sugar. Up to a point, this makes perfect sense, doesn't it?
On the other hand, there is the law of diminishing returns. What this law says is that 1 bottle of water is worth a fortune when you're thirsty in the desert but 2 bottles of water are not worth twice as much: once you're not thirsty anymore you won't value it as much, so the 1st bottle of water is worth a fortune, the 2nd bottle of water is worth a lot (since you're in the desert and there is no water to be found), the 3rd bottle is worth even less and so on. Eventually, the 87th bottle of water is worth exactly nothing to you! And the 88th bottle would require you to be paid to have the trouble of carrying it!
The same happens with time testing for defects and the number of defects found: the time invested to find the 2nd defect (in software development, in manufacturing or whatever you can think of) is a lot more then the time you require to find the 1st defect. The time required for you to find the 3rd defect is a lot more then the time it took you to find the 2nd. And so it goes. Eventually it will take you almost forever to find the 87th defect!
Time required to find defects

Now the question to be asked in the context of evaluating and selecting projects is this: what is the impact of measuring something in a linear way (say, the project's net profit)?
Lets see how important is for a project to have some profit, say 1.000€. This means that we're getting 1.000€ more in return than what we invested in the project, that is, if we invested 3,000€ we're getting 4,000€ back (which is worth 1,000€ of net profit).
Now how important is it for a project to have a net profit of 2,000€? Does it feel that a project is twice as good as the previous one if you have to invest the same money (3,000€) and you get 5,000€ back (which is a net profit of 2,000€)?
What about if you invest 2,000€ and get 4,000€ back? Do that 2,000e of net profit feel the same?
And does it feel the same as having a profit of 2,000€ but requiring an investment of 6,000€?
One thing's for sure: the answers to these questions depend on many things, including the person answering them, their risk aversion, how much money is in their pocket and so on. The one thing you can say for sure is that these are not linear: twice the net profit is not twice as good.
But what do we do in our offices? On and on we measure these in a linear scale!

BATNA & alike

When you compare your options between them you're forgetting the most basic option of them all: do nothing! It's an option that you have, but... are you considering it? And sometimes that is your best option: just do nothing. It's not always the case that your best option is the one that scored the most. One such particular example is the following.
In the context of a negotiation, people sometimes forget to check what their BATNA is (that is the Best Alternative To a Negotiated Agreement). This is really basic and it's just what your best alternative is if you don't strike a deal. Why is it so important? Because sometimes you're just better off with no deal at all. Or sometimes you're just investing to much time and energy with something for which you already have a pretty good scenario without any deal at all! An example: Suppose it costs you 1,000€ to buy some components you need for your project and you're negotiating with the manufacturing department so they make those parts. If they have to stop producing their usual stuff (worth of 5,000€) in order to produce the components you need for your project, is it worth investing time in this negotiation? It's pretty obvious you're better off buying those components for some supplier instead of building them in your manufacturing department, right? Unless you don't check first what your alternatives are!
Bottom line is this: you must always have an external option of some kind. And when you have only at stake your internal options (such as the projects that you can start) you must hear an alarm bell!
In general, what this means is that having an employee of the year means absolutely nothing. Having the best project in the company or the best programmer of the company in your team means nothing, just the same. Not if don't have some kind of benchmark: how much does it cost you to get a programmer as good as that one from outside, from the market? Being the best team member ever is no good at all if anyone you can possible hire from the market is better than that, right?! Being the best in the neighborhood (or in class) has no meaning if you can't benchmark that person with others outside: me being the fastest 100m runner in my office says nothing about how fast I run, but running 100m in 26 seconds says a lot about how fast I am - see the difference?


Sometimes we over-simplify. Like Albert Einstein once said: "make things as simple as possible, but not simpler". You should always keep this in mind, when selecting and evaluating projects - or whatever else you're doing, for that matter.
Note: Actually, Einstein said "It can scarcely be denied that the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience." and eventually someone (maybe even him) distilled it into "make things as simple as possible, but not simpler". 

Imagine you're categorizing your criteria into "good project" and "bad project". So you take the criteria on "net profit" and say that projects with more than 1,000€ of net profit are considered good projects and the others (with a net profit of 1,000€ or less) are considered bad projects. You take the criteria on ROI (Return on investment) and say that anything over 5% is good - and the rest is considered bad. You take the criteria on schedule (or duration) and say that less than 3 month is good - and the rest is bad. See the idea? So what happens when you have these projects competing to see which is the best project for the company?

Criteria for 2 projects
What we see is that are the projects are basically the same regarding net profit and ROI but that project A is much worse regarding duration. But because of the categories we built, project A is a "good project" regarding net profit and ROI, while project B is considered a "bad project". So if we average the number of "good project" that each one gets for the selected criteria, we get:

Score for 2 projects
So although we can clearly see that project B is better than project A, the truth is that we have a reasonable objective categorization process that scores project A better!
Of course I did it on purpose, I selected just 2 big categories for each criteria and attributed a net profit and a ROI just on each side of the categories. And then added a duration with a huge difference. The result: project A is better according with this objective, transparent process when it's pretty obvious its worse than project B!


Scales can change the way we perceive things. Even when things look pretty clean (objective and transparent), the fact is that our perception can be easily distorted just by the way we set up the scales. We covered 3 topics around this subject:
  • Linear scales
  • Benchmarking
  • Categorization
And once again, it was clear that we can have something 100% transparent and objective with a disastrous result!

Image from

Posted by

No comments: