Friday, May 2, 2014

Dangerous mistakes when evaluating: election methods


When evaluating something (selecting a supplier or which projects will get to start next month), bias are almost sure to occur.
But the worst kind of errors that can happen are those that we provoke when trying to make things more objective. We may believe we have a close to perfect process to select a supplier because it is based on an election process when in fact the result depends on what method we used to select the lucky winner.





An example

Suppose we have to choose what the is best project for an organization and that this particular organization has 5 candidate projects, let us say projects A, B, C, D, and E. As we have a pool of 25 project management experts, we asked them to select what the best project is. Easy, right? The most voted project should be the best project for this organization and so that should be where this organization should put their money on.
Please note: The selection criteria is not the focus of this article, so you can consider that the best project is a weighted combination of the return on investment, payback period and overall risk - or any other criteria. It is all the same as this is not what we're discussing on this article, OK?
Now, suppose we ask each of these 25 Project Management experts to rank the projects from best to worst. Suppose these were the results:
Results

So if you look at the 2nd column, you should read it like this: 8 people/voters found the best project to be project A, then C, then D, then B, and the worst was project E. And the 3rd column reads: 7 people/voters found the best project to be project B, then D, then C, then E and finally A. Pretty straight forward, right?
And it also seems fair, doesn't it?

What project should be selected?

So let the fun begin and answer this simple question: what project should be selected?

Project A should be selected

The project that was voted the most is project A so project A should be selected - 8 people voted on project A, 7 on B, 2 on C, and 4 on D and E. The project with more votes is project A, and so project A should be selected.
But then you notice that project A was considered the worst project by all the other voters. You find this odd and end up thinking that you have a 2nd round. 

No, project B should be selected

So you select the 2 most voted projects (project A with 8 votes and project B with 7 votes) and eliminate all other projects from this election.
Results with a 2nd round
Now project A gets 8 votes and project B is selected with 7+4+4+2 = 17 votes - even more votes than project A got on the 1st round! Is this getting weirder and weirder or what? How can we have a clear winner with an election method and another clear winner if we use some other election method? Maybe these methods are not good for project selection. Maybe you should try some other method...
By the way, this method is also a classic election method called Instant-runoff voting.

Or is it project E?

Let's try another method, this time eliminating the worst project again and again and see what is the project left - that should clearly be the best project, right? So this classic method works like this:

  1. Count the total votes for each candidate (we have project A with 8 votes, B with 7, C with 2, and D and E with 4 votes)
  2. We throw away the candidate with the least votes (so we lose project C and keep projects A, B, D and E)
  3. Results after eliminating C

  4. Now we repeat the process (step 1 again) and count the votes for each project (and we now have project A with 8 votes, B with 7, D with 4, and project E with 4+2 = 6 votes)
  5. We throw away the candidate with the least votes (so we lose project D and keep projects A, B and E)
  6. Results after eliminating D
  7. So counting the votes again we have project A with 8 votes, B with 7, and project E with 4+4+2 = 10 votes
  8. So we repeat the process again and lose the least voted project (which is now project B)
  9. Recount the votes (A with 8 and E with 7+4+4+2 = 17 votes)
  10. And we have a new winner: project E
It looks like the election method we chose is determining who the winner is, isn't it? Or maybe this method is no good as well. Maybe we should try a more objective approach. Maybe we should pick...

...the most consistently voted

This seems better: we can assign points to each position and check the total points for each project. And the project with more points should be our lucky winner. Fair and square, right? In fact, this method even has a name - it is called the Borda method. As we have 5 possible options we'll score the 1st choice with 5 points, the 2nd with 4 points , and so on till the last choice is awarded 1 point.

Results with awarded points
So project A get a total of 8x5 points + 7x1 points + 4x1 points + 4x1 points + 2x1 points = 57 points. If you compute the points awarded to the other projects, you get:

Total points awarded
And so the most consistently voted project is... not project A, not B, not E, but in fact it is project D!

What is going on?

The truth is that having an objective way to evaluate options is not enough. You can in fact start off with that, but that doesn't make the process objective, fair and consistent. Having a weighted scoring grid like an Analytic Hierarchy based grid (ready to use and available to download here) is not enough.
The first dangerous error is thinking that, because you have an objective approach, the results are also objective - and fair and consistent. As this example clearly shows, and although all of these different methods make sense, they all yield different results. The method you chose determines the result you get!
And although this voting grid was constructed on purpose to yield different results according to which of these 4 classic voting methods was used, the reality is that there is a huge number of different methods to do it. Just check the wikipedia!
And this is just one of these dangerous mistakes (dangerous because we believe that we're making an objective, fair and consistent evaluation), there are others! Keep tuned because I'll probably write about them next...


Image from http://www.learnersonline.com/

Posted by

No comments: