Do your experts make very different judgements given the same information?
Or does a single expert judge things differently depending on their mood?
If so, they are not unusual, there is “noise” in human judgement.
Here red10 ‘s Sarah Barber describes three strategies to help teams reduce noise and make better judgements.
What is “noise”?
In his book Noise, Daniel Kahneman describes research in many fields – including medicine, law and forensic science which shows that experts can make very different judgements based on the same information.
One example that he gives concerns wildly differing sentencing decisions – a particularly problematic area as it leads to inequity in the judicial system. Not only do these decisions vary between judges, but the same judge may be more lenient after lunch than they are first thing in the morning.
He calls this variation noise because it is random error. This distinguishes it from the systematic error that bias would cause (e.g. being particularly lenient towards one racial group).
In this example, the use of rules or guidelines substantially reduces noise – and this is also true in medicine.
Here are three strategies that teams can use to reduce noise.
Strategy 1 – The wisdom of crowds
In a traditional game at UK village fetes, participants are asked to guess the number of sweets in a jar. The closest guess wins the jar. It turns out that many people make inaccurate guesses, but just taking a simple average of those guesses tends to give an answer that is close to the true value. This phenomenon is called “the wisdom of crowds”.
It is a method that we at red10 use with teams. When collecting feedback from team members on the effectiveness of their team, we ask them to give a score on nine dimensions of team effectiveness. The results are fed back to the team as an average and a range, preserving the anonymity of individual scorers. In our experience, this average is a pretty good indicator of how well the team is performing on a particular dimension. And where there is a wide range (or disparity) in scores, team discussion about the reasons for this can be very fruitful in identifying improvements the team would like to make.
This collection of anonymised individual judgements before the discussion reduces noise because it avoids cascade effects (everyone getting behind one individual, which biases the outcome) and group polarisation. It can equally well be applied to the business decisions that teams need to make.
Strategy 2 – Practise active open-mindedness
Kahneman describes a large piece of research which he called the “good judgement project” in which many individuals were asked to forecast outcomes based on a set of data. He found that about 2% of the volunteers on the project performed much better than everyone else, and he dubbed these people “super-forecasters”. It is possible to learn from what these people do.
Super-forecasters demonstrated what Kahneman called active open-mindedness. They considered evidence that went against their beliefs and paid attention to disagreement.
One tool that a team can use to help them pay attention to disagreement is called “fist five” . If you appear to be reaching consensus on a decision, check by asking everyone to hold up their hand with a number of fingers indicating the strength of their support for the decision. It is important that everyone raises their hand at the same time. Five fingers in the air means fully support, a fist or just one or two fingers means the person does not support, or has reservations. Then, listen to what the person disagreeing with the decision has to say. This technique helps dissenters to find the courage to speak up and add their expertise to the discussion. You may want to modify your decision or proposal in the light of what they say.
Strategy 3 – Break down the complex into independent fact-based assessments
If you can break down a complex judgement into multiple, fact-based assessments you reduce the risk of excessive coherence. In other words, you minimise the risk of people distorting or ignoring information that doesn’t support an emerging story.
Many people will be familiar with the use of structured interviews for recruitment. In these interviews people are assessed on a set of competencies relevant to the job. Each assessment is discussed separately before hiring decisions are made.
A cross-functional team of experts is well set up to work in this way. Kahneman gives the example of a deal-making team. In the analytical phase, assessments should be kept as independent as possible. In the decision meeting, review each assessment separately. Intuition should be delayed until you make the final decision, that way it will be anchored in facts-based assessments.